Oct 30 21:27:39.320011 ip-10-0-140-163 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:27:39.701465 ip-10-0-140-163 kubenswrapper[2543]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:27:39.701465 ip-10-0-140-163 kubenswrapper[2543]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:27:39.701465 ip-10-0-140-163 kubenswrapper[2543]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:27:39.701465 ip-10-0-140-163 kubenswrapper[2543]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:27:39.706715 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706639 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:27:39.706715 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706716 2543 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:27:39.706796 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706720 2543 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:27:39.706796 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706723 2543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:27:39.706796 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706726 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:27:39.706796 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706729 2543 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:27:39.706796 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706732 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:27:39.706796 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706735 2543 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:27:39.706796 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706737 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:27:39.706796 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706763 2543 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:27:39.706796 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706768 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:27:39.706796 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706772 2543 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:27:39.706796 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706776 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:27:39.706796 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706779 2543 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:27:39.706796 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706782 2543 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:27:39.706796 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706785 2543 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:27:39.706796 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706788 2543 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:27:39.706796 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706792 2543 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:27:39.706796 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706797 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:27:39.706796 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706800 2543 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:27:39.706796 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706804 2543 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:27:39.707227 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706807 2543 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:27:39.707227 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706810 2543 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:27:39.707227 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706813 2543 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:27:39.707227 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706815 2543 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:27:39.707227 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706818 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:27:39.707227 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706821 2543 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:27:39.707227 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706823 2543 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:27:39.707227 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706826 2543 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:27:39.707227 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706829 2543 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:27:39.707227 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706831 2543 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:27:39.707227 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706834 2543 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:27:39.707227 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706836 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:27:39.707227 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706839 2543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:27:39.707227 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706841 2543 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:27:39.707227 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706844 2543 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:27:39.707227 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706847 2543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:27:39.707227 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706849 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:27:39.707227 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706852 2543 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:27:39.707227 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706855 2543 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:27:39.707227 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706857 2543 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:27:39.707683 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706860 2543 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:27:39.707683 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706864 2543 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:27:39.707683 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706867 2543 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:27:39.707683 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706871 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:27:39.707683 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706874 2543 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:27:39.707683 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706876 2543 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:27:39.707683 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706879 2543 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:27:39.707683 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706882 2543 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:27:39.707683 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706884 2543 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:27:39.707683 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706887 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:27:39.707683 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706890 2543 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:27:39.707683 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706893 2543 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:27:39.707683 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706895 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:27:39.707683 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706898 2543 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:27:39.707683 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706901 2543 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:27:39.707683 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706904 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:27:39.707683 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706907 2543 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:27:39.707683 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706909 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:27:39.707683 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706912 2543 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:27:39.708134 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706916 2543 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:27:39.708134 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706919 2543 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:27:39.708134 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706921 2543 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:27:39.708134 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706924 2543 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:27:39.708134 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706926 2543 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:27:39.708134 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706929 2543 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:27:39.708134 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706932 2543 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:27:39.708134 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706934 2543 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:27:39.708134 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706937 2543 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:27:39.708134 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706940 2543 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:27:39.708134 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706942 2543 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:27:39.708134 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706945 2543 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:27:39.708134 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706948 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:27:39.708134 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706951 2543 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:27:39.708134 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706953 2543 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:27:39.708134 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706956 2543 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:27:39.708134 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706959 2543 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:27:39.708134 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706961 2543 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:27:39.708134 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706965 2543 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:27:39.708134 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706968 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:27:39.708592 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706971 2543 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:27:39.708592 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706974 2543 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:27:39.708592 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706976 2543 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:27:39.708592 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706979 2543 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:27:39.708592 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706982 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:27:39.708592 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706984 2543 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:27:39.708592 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706987 2543 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:27:39.708592 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706989 2543 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:27:39.708592 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706992 2543 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:27:39.708592 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706995 2543 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:27:39.708592 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.706998 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:27:39.708592 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.707000 2543 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:27:39.708592 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.707003 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:27:39.708592 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.707005 2543 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:27:39.708592 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.707008 2543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:27:39.708592 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.707010 2543 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:27:39.708592 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.707013 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:27:39.708592 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.707015 2543 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:27:39.708592 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.707018 2543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:27:39.708592 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.707020 2543 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:27:39.709044 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.707024 2543 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:27:39.709044 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.707026 2543 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:27:39.709044 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.707029 2543 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:27:39.709044 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.707032 2543 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:27:39.709044 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.707034 2543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:27:39.709044 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.707037 2543 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:27:39.709887 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709875 2543 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:27:39.709887 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709887 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:27:39.709949 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709891 2543 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:27:39.709949 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709894 2543 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:27:39.709949 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709897 2543 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:27:39.709949 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709900 2543 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:27:39.709949 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709903 2543 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:27:39.709949 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709905 2543 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:27:39.709949 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709908 2543 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:27:39.709949 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709911 2543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:27:39.709949 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709913 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:27:39.709949 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709915 2543 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:27:39.709949 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709918 2543 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:27:39.709949 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709921 2543 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:27:39.709949 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709923 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:27:39.709949 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709925 2543 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:27:39.709949 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709928 2543 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:27:39.709949 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709931 2543 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:27:39.709949 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709934 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:27:39.709949 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709936 2543 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:27:39.709949 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709938 2543 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:27:39.709949 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709941 2543 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:27:39.710426 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709944 2543 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:27:39.710426 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709946 2543 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:27:39.710426 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709948 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:27:39.710426 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709951 2543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:27:39.710426 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709954 2543 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:27:39.710426 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709957 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:27:39.710426 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709960 2543 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:27:39.710426 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709963 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:27:39.710426 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709965 2543 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:27:39.710426 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709968 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:27:39.710426 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709971 2543 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:27:39.710426 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709974 2543 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:27:39.710426 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709976 2543 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:27:39.710426 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709979 2543 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:27:39.710426 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709981 2543 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:27:39.710426 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709984 2543 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:27:39.710426 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709986 2543 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:27:39.710426 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709989 2543 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:27:39.710426 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709991 2543 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:27:39.710426 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709994 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:27:39.710923 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709996 2543 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:27:39.710923 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.709999 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:27:39.710923 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710001 2543 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:27:39.710923 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710004 2543 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:27:39.710923 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710006 2543 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:27:39.710923 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710009 2543 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:27:39.710923 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710011 2543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:27:39.710923 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710014 2543 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:27:39.710923 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710016 2543 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:27:39.710923 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710019 2543 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:27:39.710923 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710021 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:27:39.710923 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710023 2543 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:27:39.710923 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710026 2543 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:27:39.710923 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710029 2543 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:27:39.710923 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710031 2543 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:27:39.710923 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710034 2543 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:27:39.710923 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710037 2543 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:27:39.710923 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710040 2543 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:27:39.710923 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710042 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:27:39.711369 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710044 2543 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:27:39.711369 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710047 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:27:39.711369 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710049 2543 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:27:39.711369 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710052 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:27:39.711369 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710055 2543 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:27:39.711369 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710058 2543 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:27:39.711369 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710060 2543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:27:39.711369 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710063 2543 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:27:39.711369 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710065 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:27:39.711369 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710068 2543 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:27:39.711369 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710070 2543 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:27:39.711369 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710073 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:27:39.711369 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710075 2543 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:27:39.711369 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710078 2543 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:27:39.711369 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710081 2543 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:27:39.711369 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710083 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:27:39.711369 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710086 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:27:39.711369 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710088 2543 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:27:39.711369 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710091 2543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:27:39.711824 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710093 2543 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:27:39.711824 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710096 2543 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:27:39.711824 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710099 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:27:39.711824 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710101 2543 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:27:39.711824 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710104 2543 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:27:39.711824 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710106 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:27:39.711824 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710110 2543 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:27:39.711824 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710113 2543 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:27:39.711824 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710116 2543 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:27:39.711824 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710118 2543 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:27:39.711824 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710121 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:27:39.711824 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710125 2543 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:27:39.711824 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710129 2543 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:27:39.711824 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710132 2543 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:27:39.711824 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710134 2543 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:27:39.711824 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710137 2543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:27:39.711824 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710140 2543 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:27:39.711824 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710143 2543 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:27:39.711824 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710146 2543 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:27:39.712255 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710149 2543 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:27:39.712255 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710154 2543 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:27:39.712255 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710158 2543 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:27:39.712255 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710160 2543 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:27:39.712255 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710163 2543 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:27:39.712255 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710166 2543 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:27:39.712255 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710168 2543 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:27:39.712255 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710691 2543 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:27:39.712255 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710697 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:27:39.712255 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710700 2543 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:27:39.712255 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710703 2543 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:27:39.712255 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710706 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:27:39.712255 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710709 2543 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:27:39.712255 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710712 2543 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:27:39.712255 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710715 2543 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:27:39.712255 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710717 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:27:39.712255 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710720 2543 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:27:39.712255 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710723 2543 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:27:39.712255 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710725 2543 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:27:39.712698 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710728 2543 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:27:39.712698 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710731 2543 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:27:39.712698 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710734 2543 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:27:39.712698 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710736 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:27:39.712698 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710739 2543 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:27:39.712698 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710741 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:27:39.712698 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710745 2543 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:27:39.712698 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710747 2543 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:27:39.712698 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710767 2543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:27:39.712698 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710772 2543 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:27:39.712698 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710775 2543 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:27:39.712698 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710777 2543 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:27:39.712698 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710781 2543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:27:39.712698 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710783 2543 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:27:39.712698 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710786 2543 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:27:39.712698 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710789 2543 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:27:39.712698 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710792 2543 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:27:39.712698 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710794 2543 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:27:39.712698 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710797 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:27:39.712698 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710799 2543 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:27:39.713168 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710802 2543 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:27:39.713168 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710805 2543 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:27:39.713168 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710807 2543 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:27:39.713168 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710810 2543 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:27:39.713168 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710813 2543 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:27:39.713168 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710816 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:27:39.713168 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710818 2543 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:27:39.713168 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710822 2543 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:27:39.713168 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710826 2543 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:27:39.713168 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710829 2543 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:27:39.713168 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710833 2543 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:27:39.713168 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710836 2543 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:27:39.713168 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710839 2543 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:27:39.713168 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710842 2543 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:27:39.713168 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710845 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:27:39.713168 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710847 2543 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:27:39.713168 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710850 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:27:39.713168 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710853 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:27:39.713168 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710855 2543 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:27:39.713168 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710858 2543 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:27:39.713628 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710860 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:27:39.713628 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710863 2543 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:27:39.713628 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710865 2543 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:27:39.713628 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710868 2543 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:27:39.713628 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710870 2543 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:27:39.713628 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710874 2543 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:27:39.713628 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710876 2543 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:27:39.713628 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710879 2543 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:27:39.713628 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710882 2543 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:27:39.713628 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710884 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:27:39.713628 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710886 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:27:39.713628 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710889 2543 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:27:39.713628 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710892 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:27:39.713628 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710894 2543 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:27:39.713628 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710897 2543 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:27:39.713628 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710899 2543 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:27:39.713628 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710902 2543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:27:39.713628 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710905 2543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:27:39.713628 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710907 2543 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:27:39.713628 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710910 2543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:27:39.714105 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710912 2543 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:27:39.714105 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710915 2543 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:27:39.714105 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710918 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:27:39.714105 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710920 2543 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:27:39.714105 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710923 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:27:39.714105 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710925 2543 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:27:39.714105 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710928 2543 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:27:39.714105 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710931 2543 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:27:39.714105 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710933 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:27:39.714105 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710936 2543 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:27:39.714105 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710939 2543 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:27:39.714105 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710941 2543 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:27:39.714105 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710944 2543 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:27:39.714105 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710946 2543 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:27:39.714105 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710948 2543 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:27:39.714105 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710951 2543 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:27:39.714105 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710953 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:27:39.714105 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710956 2543 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:27:39.714105 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710959 2543 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:27:39.714582 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710961 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:27:39.714582 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710964 2543 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:27:39.714582 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710966 2543 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:27:39.714582 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710969 2543 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:27:39.714582 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710971 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:27:39.714582 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710974 2543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:27:39.714582 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710976 2543 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:27:39.714582 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710979 2543 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:27:39.714582 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710981 2543 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:27:39.714582 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710986 2543 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:27:39.714582 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710990 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:27:39.714582 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710994 2543 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:27:39.714582 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710996 2543 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:27:39.714582 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.710999 2543 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:27:39.714582 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.711001 2543 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:27:39.714582 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712056 2543 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:27:39.714582 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712065 2543 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:27:39.714582 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712072 2543 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:27:39.714582 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712077 2543 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:27:39.714582 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712081 2543 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:27:39.715067 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712084 2543 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:27:39.715067 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712089 2543 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:27:39.715067 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712093 2543 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:27:39.715067 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712096 2543 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:27:39.715067 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712099 2543 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:27:39.715067 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712102 2543 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:27:39.715067 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712105 2543 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:27:39.715067 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712109 2543 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:27:39.715067 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712111 2543 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:27:39.715067 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712114 2543 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:27:39.715067 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712117 2543 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:27:39.715067 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712120 2543 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:27:39.715067 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712124 2543 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:27:39.715067 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712128 2543 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:27:39.715067 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712131 2543 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:27:39.715067 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712134 2543 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:27:39.715067 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712137 2543 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:27:39.715067 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712141 2543 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:27:39.715067 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712145 2543 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:27:39.715067 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712147 2543 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:27:39.715067 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712151 2543 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:27:39.715067 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712154 2543 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:27:39.715067 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712157 2543 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:27:39.715067 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712161 2543 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712163 2543 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712167 2543 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712170 2543 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712174 2543 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712177 2543 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712180 2543 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712183 2543 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712186 2543 flags.go:64] FLAG: --enable-server="true" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712189 2543 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712194 2543 flags.go:64] FLAG: --event-burst="100" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712197 2543 flags.go:64] FLAG: --event-qps="50" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712200 2543 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712203 2543 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712206 2543 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712210 2543 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712214 2543 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712217 2543 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712220 2543 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712223 2543 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712226 2543 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712230 2543 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712233 2543 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712236 2543 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712239 2543 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:27:39.715617 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712242 2543 flags.go:64] FLAG: --feature-gates="" Oct 30 21:27:39.716235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712246 2543 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:27:39.716235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712249 2543 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:27:39.716235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712252 2543 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:27:39.716235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712256 2543 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:27:39.716235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712259 2543 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:27:39.716235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712265 2543 flags.go:64] FLAG: --help="false" Oct 30 21:27:39.716235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712268 2543 flags.go:64] FLAG: --hostname-override="ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:27:39.716235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712272 2543 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:27:39.716235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712275 2543 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:27:39.716235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712278 2543 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:27:39.716235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712282 2543 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:27:39.716235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712285 2543 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:27:39.716235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712288 2543 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:27:39.716235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712291 2543 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:27:39.716235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712294 2543 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:27:39.716235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712297 2543 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:27:39.716235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712300 2543 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:27:39.716235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712303 2543 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:27:39.716235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712306 2543 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:27:39.716235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712308 2543 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:27:39.716235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712312 2543 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:27:39.716235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712314 2543 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:27:39.716235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712319 2543 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712322 2543 flags.go:64] FLAG: --lock-file="" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712325 2543 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712328 2543 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712331 2543 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712336 2543 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712339 2543 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712343 2543 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712346 2543 flags.go:64] FLAG: --logging-format="text" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712349 2543 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712352 2543 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712355 2543 flags.go:64] FLAG: --manifest-url="" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712358 2543 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712365 2543 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712368 2543 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712373 2543 flags.go:64] FLAG: --max-pods="110" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712376 2543 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712379 2543 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712383 2543 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712386 2543 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712389 2543 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712392 2543 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712395 2543 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712401 2543 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712405 2543 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:27:39.716806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712408 2543 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712411 2543 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712414 2543 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712416 2543 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712420 2543 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712422 2543 flags.go:64] FLAG: --port="10250" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712425 2543 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712430 2543 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0ed923fb3a7a56b27" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712433 2543 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712436 2543 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712439 2543 flags.go:64] FLAG: --register-node="true" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712442 2543 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712446 2543 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712449 2543 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712451 2543 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712455 2543 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712459 2543 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712462 2543 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712465 2543 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712467 2543 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712470 2543 flags.go:64] FLAG: --runonce="false" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712473 2543 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712478 2543 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712481 2543 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712484 2543 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712487 2543 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:27:39.717386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712490 2543 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:27:39.718041 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712493 2543 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:27:39.718041 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712496 2543 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:27:39.718041 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712499 2543 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:27:39.718041 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712502 2543 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:27:39.718041 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712505 2543 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:27:39.718041 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712508 2543 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:27:39.718041 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712511 2543 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:27:39.718041 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712514 2543 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:27:39.718041 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712517 2543 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:27:39.718041 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712522 2543 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:27:39.718041 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712525 2543 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:27:39.718041 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712528 2543 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:27:39.718041 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712533 2543 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:27:39.718041 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712536 2543 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:27:39.718041 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712539 2543 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:27:39.718041 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712542 2543 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:27:39.718041 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712545 2543 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:27:39.718041 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712548 2543 flags.go:64] FLAG: --v="2" Oct 30 21:27:39.718041 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712552 2543 flags.go:64] FLAG: --version="false" Oct 30 21:27:39.718041 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712556 2543 flags.go:64] FLAG: --vmodule="" Oct 30 21:27:39.718041 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712560 2543 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:27:39.718041 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.712564 2543 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:27:39.718041 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712695 2543 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:27:39.718041 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712698 2543 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:27:39.718608 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712702 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:27:39.718608 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712705 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:27:39.718608 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712708 2543 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:27:39.718608 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712712 2543 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:27:39.718608 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712715 2543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:27:39.718608 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712718 2543 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:27:39.718608 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712720 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:27:39.718608 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712723 2543 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:27:39.718608 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712726 2543 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:27:39.718608 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712728 2543 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:27:39.718608 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712731 2543 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:27:39.718608 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712733 2543 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:27:39.718608 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712736 2543 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:27:39.718608 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712738 2543 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:27:39.718608 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712741 2543 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:27:39.718608 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712744 2543 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:27:39.718608 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712746 2543 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:27:39.718608 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712760 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:27:39.718608 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712763 2543 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:27:39.718608 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712765 2543 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:27:39.719143 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712770 2543 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:27:39.719143 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712773 2543 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:27:39.719143 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712776 2543 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:27:39.719143 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712778 2543 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:27:39.719143 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712781 2543 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:27:39.719143 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712784 2543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:27:39.719143 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712786 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:27:39.719143 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712789 2543 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:27:39.719143 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712792 2543 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:27:39.719143 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712794 2543 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:27:39.719143 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712797 2543 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:27:39.719143 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712800 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:27:39.719143 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712803 2543 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:27:39.719143 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712805 2543 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:27:39.719143 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712807 2543 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:27:39.719143 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712811 2543 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:27:39.719143 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712815 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:27:39.719143 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712817 2543 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:27:39.719143 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712820 2543 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:27:39.719143 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712822 2543 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:27:39.719618 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712825 2543 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:27:39.719618 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712828 2543 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:27:39.719618 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712830 2543 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:27:39.719618 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712833 2543 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:27:39.719618 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712836 2543 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:27:39.719618 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712838 2543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:27:39.719618 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712841 2543 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:27:39.719618 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712843 2543 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:27:39.719618 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712846 2543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:27:39.719618 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712849 2543 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:27:39.719618 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712851 2543 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:27:39.719618 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712854 2543 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:27:39.719618 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712858 2543 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:27:39.719618 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712861 2543 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:27:39.719618 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712863 2543 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:27:39.719618 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712866 2543 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:27:39.719618 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712868 2543 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:27:39.719618 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712871 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:27:39.719618 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712873 2543 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:27:39.719618 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712876 2543 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:27:39.720088 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712878 2543 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:27:39.720088 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712881 2543 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:27:39.720088 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712884 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:27:39.720088 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712887 2543 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:27:39.720088 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712889 2543 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:27:39.720088 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712892 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:27:39.720088 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712894 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:27:39.720088 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712898 2543 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:27:39.720088 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712900 2543 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:27:39.720088 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712903 2543 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:27:39.720088 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712906 2543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:27:39.720088 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712910 2543 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:27:39.720088 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712913 2543 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:27:39.720088 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712917 2543 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:27:39.720088 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712920 2543 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:27:39.720088 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712922 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:27:39.720088 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712925 2543 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:27:39.720088 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712927 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:27:39.720088 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712931 2543 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:27:39.720529 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712934 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:27:39.720529 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712938 2543 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:27:39.720529 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712941 2543 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:27:39.720529 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712944 2543 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:27:39.720529 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712947 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:27:39.720529 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712951 2543 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:27:39.720529 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712954 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:27:39.720529 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712956 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:27:39.720529 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712959 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:27:39.720529 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712962 2543 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:27:39.720529 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712965 2543 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:27:39.720529 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712967 2543 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:27:39.720529 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712970 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:27:39.720529 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712972 2543 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:27:39.720529 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712975 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:27:39.720529 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712977 2543 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:27:39.720529 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712980 2543 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:27:39.720529 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712984 2543 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:27:39.720529 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712986 2543 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:27:39.720529 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712989 2543 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:27:39.721003 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712992 2543 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:27:39.721003 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712995 2543 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:27:39.721003 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.712997 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:27:39.721003 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713000 2543 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:27:39.721003 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713003 2543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:27:39.721003 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.713028 2543 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:27:39.721003 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713063 2543 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:27:39.721003 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713068 2543 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:27:39.721003 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713071 2543 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:27:39.721003 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713074 2543 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:27:39.721003 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713077 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:27:39.721003 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713080 2543 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:27:39.721003 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713082 2543 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:27:39.721003 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713085 2543 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:27:39.721003 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713088 2543 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:27:39.721003 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713090 2543 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:27:39.721371 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713093 2543 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:27:39.721371 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713095 2543 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:27:39.721371 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713099 2543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:27:39.721371 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713102 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:27:39.721371 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713104 2543 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:27:39.721371 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713107 2543 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:27:39.721371 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713110 2543 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:27:39.721371 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713112 2543 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:27:39.721371 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713115 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:27:39.721371 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713117 2543 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:27:39.721371 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713120 2543 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:27:39.721371 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713123 2543 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:27:39.721371 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713125 2543 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:27:39.721371 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713128 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:27:39.721371 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713131 2543 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:27:39.721371 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713134 2543 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:27:39.721371 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713138 2543 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:27:39.721371 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713140 2543 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:27:39.721371 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713143 2543 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:27:39.721937 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713146 2543 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:27:39.721937 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713148 2543 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:27:39.721937 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713150 2543 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:27:39.721937 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713153 2543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:27:39.721937 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713155 2543 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:27:39.721937 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713158 2543 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:27:39.721937 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713161 2543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:27:39.721937 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713164 2543 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:27:39.721937 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713166 2543 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:27:39.721937 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713168 2543 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:27:39.721937 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713171 2543 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:27:39.721937 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713174 2543 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:27:39.721937 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713177 2543 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:27:39.721937 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713180 2543 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:27:39.721937 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713182 2543 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:27:39.721937 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713186 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:27:39.721937 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713188 2543 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:27:39.721937 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713191 2543 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:27:39.721937 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713193 2543 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:27:39.721937 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713196 2543 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:27:39.722398 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713198 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:27:39.722398 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713201 2543 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:27:39.722398 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713203 2543 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:27:39.722398 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713206 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:27:39.722398 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713208 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:27:39.722398 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713210 2543 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:27:39.722398 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713213 2543 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:27:39.722398 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713215 2543 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:27:39.722398 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713218 2543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:27:39.722398 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713223 2543 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:27:39.722398 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713227 2543 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:27:39.722398 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713230 2543 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:27:39.722398 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713232 2543 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:27:39.722398 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713235 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:27:39.722398 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713238 2543 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:27:39.722398 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713240 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:27:39.722398 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713245 2543 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:27:39.722398 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713248 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:27:39.722398 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713252 2543 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:27:39.722863 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713254 2543 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:27:39.722863 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713257 2543 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:27:39.722863 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713259 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:27:39.722863 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713262 2543 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:27:39.722863 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713264 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:27:39.722863 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713267 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:27:39.722863 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713270 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:27:39.722863 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713272 2543 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:27:39.722863 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713275 2543 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:27:39.722863 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713277 2543 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:27:39.722863 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713279 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:27:39.722863 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713282 2543 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:27:39.722863 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713284 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:27:39.722863 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713287 2543 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:27:39.722863 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713289 2543 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:27:39.722863 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713292 2543 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:27:39.722863 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713294 2543 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:27:39.722863 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713297 2543 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:27:39.722863 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713299 2543 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:27:39.723334 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713302 2543 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:27:39.723334 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713305 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:27:39.723334 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713307 2543 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:27:39.723334 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713310 2543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:27:39.723334 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713313 2543 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:27:39.723334 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713315 2543 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:27:39.723334 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713318 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:27:39.723334 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713320 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:27:39.723334 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713323 2543 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:27:39.723334 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713326 2543 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:27:39.723334 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713328 2543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:27:39.723334 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713331 2543 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:27:39.723334 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713333 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:27:39.723334 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713336 2543 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:27:39.723334 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713338 2543 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:27:39.723334 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713341 2543 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:27:39.723334 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713343 2543 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:27:39.723334 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713346 2543 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:27:39.723334 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.713349 2543 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:27:39.723334 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.720405 2543 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:27:39.723818 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.720419 2543 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:27:39.723818 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.720434 2543 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:27:39.723818 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.720439 2543 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:27:39.723818 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720492 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:27:39.723818 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720497 2543 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:27:39.723818 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720500 2543 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:27:39.723818 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720504 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:27:39.723818 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720507 2543 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:27:39.723818 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720510 2543 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:27:39.723818 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720512 2543 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:27:39.723818 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720515 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:27:39.723818 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720518 2543 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:27:39.723818 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720521 2543 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:27:39.723818 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720523 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:27:39.723818 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720526 2543 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:27:39.723818 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720528 2543 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:27:39.723818 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720531 2543 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:27:39.723818 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720534 2543 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:27:39.723818 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720538 2543 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:27:39.724257 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720542 2543 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:27:39.724257 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720545 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:27:39.724257 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720548 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:27:39.724257 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720551 2543 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:27:39.724257 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720554 2543 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:27:39.724257 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720558 2543 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:27:39.724257 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720561 2543 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:27:39.724257 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720564 2543 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:27:39.724257 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720567 2543 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:27:39.724257 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720569 2543 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:27:39.724257 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720572 2543 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:27:39.724257 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720575 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:27:39.724257 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720579 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:27:39.724257 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720582 2543 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:27:39.724257 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720584 2543 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:27:39.724257 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720587 2543 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:27:39.724257 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720589 2543 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:27:39.724257 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720592 2543 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:27:39.724257 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720594 2543 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:27:39.724699 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720597 2543 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:27:39.724699 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720599 2543 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:27:39.724699 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720602 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:27:39.724699 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720604 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:27:39.724699 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720607 2543 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:27:39.724699 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720609 2543 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:27:39.724699 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720612 2543 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:27:39.724699 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720614 2543 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:27:39.724699 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720617 2543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:27:39.724699 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720619 2543 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:27:39.724699 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720622 2543 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:27:39.724699 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720625 2543 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:27:39.724699 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720627 2543 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:27:39.724699 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720630 2543 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:27:39.724699 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720632 2543 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:27:39.724699 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720635 2543 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:27:39.724699 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720637 2543 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:27:39.724699 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720640 2543 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:27:39.724699 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720642 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:27:39.725181 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720645 2543 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:27:39.725181 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720647 2543 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:27:39.725181 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720650 2543 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:27:39.725181 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720652 2543 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:27:39.725181 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720655 2543 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:27:39.725181 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720658 2543 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:27:39.725181 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720661 2543 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:27:39.725181 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720663 2543 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:27:39.725181 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720666 2543 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:27:39.725181 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720669 2543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:27:39.725181 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720671 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:27:39.725181 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720674 2543 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:27:39.725181 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720676 2543 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:27:39.725181 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720679 2543 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:27:39.725181 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720681 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:27:39.725181 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720684 2543 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:27:39.725181 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720686 2543 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:27:39.725181 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720689 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:27:39.725181 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720691 2543 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:27:39.725631 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720694 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:27:39.725631 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720696 2543 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:27:39.725631 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720699 2543 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:27:39.725631 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720701 2543 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:27:39.725631 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720704 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:27:39.725631 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720706 2543 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:27:39.725631 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720708 2543 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:27:39.725631 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720711 2543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:27:39.725631 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720713 2543 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:27:39.725631 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720716 2543 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:27:39.725631 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720718 2543 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:27:39.725631 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720721 2543 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:27:39.725631 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720723 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:27:39.725631 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720726 2543 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:27:39.725631 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720729 2543 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:27:39.725631 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720732 2543 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:27:39.725631 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720734 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:27:39.725631 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720737 2543 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:27:39.725631 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720740 2543 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:27:39.725631 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720742 2543 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:27:39.726148 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720745 2543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:27:39.726148 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720747 2543 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:27:39.726148 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720768 2543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:27:39.726148 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720771 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:27:39.726148 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720773 2543 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:27:39.726148 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720776 2543 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:27:39.726148 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720779 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:27:39.726148 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720781 2543 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:27:39.726148 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720784 2543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:27:39.726148 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720786 2543 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:27:39.726148 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720789 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:27:39.726148 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720792 2543 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:27:39.726148 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720794 2543 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:27:39.726148 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.720822 2543 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:27:39.726148 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720970 2543 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:27:39.726148 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720975 2543 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:27:39.726527 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720978 2543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:27:39.726527 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720981 2543 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:27:39.726527 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720984 2543 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:27:39.726527 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720987 2543 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:27:39.726527 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720991 2543 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:27:39.726527 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720994 2543 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:27:39.726527 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720996 2543 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:27:39.726527 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.720999 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:27:39.726527 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721001 2543 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:27:39.726527 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721004 2543 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:27:39.726527 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721007 2543 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:27:39.726527 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721010 2543 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:27:39.726527 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721012 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:27:39.726527 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721015 2543 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:27:39.726527 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721018 2543 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:27:39.726527 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721020 2543 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:27:39.726527 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721023 2543 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:27:39.726527 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721026 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:27:39.726527 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721028 2543 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:27:39.727674 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721032 2543 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:27:39.727674 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721035 2543 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:27:39.727674 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721037 2543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:27:39.727674 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721040 2543 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:27:39.727674 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721042 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:27:39.727674 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721045 2543 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:27:39.727674 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721047 2543 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:27:39.727674 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721050 2543 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:27:39.727674 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721052 2543 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:27:39.727674 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721054 2543 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:27:39.727674 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721057 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:27:39.727674 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721059 2543 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:27:39.727674 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721062 2543 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:27:39.727674 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721064 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:27:39.727674 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721067 2543 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:27:39.727674 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721070 2543 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:27:39.727674 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721072 2543 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:27:39.727674 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721075 2543 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:27:39.727674 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721078 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:27:39.728170 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721080 2543 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:27:39.728170 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721083 2543 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:27:39.728170 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721085 2543 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:27:39.728170 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721088 2543 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:27:39.728170 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721090 2543 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:27:39.728170 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721093 2543 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:27:39.728170 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721095 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:27:39.728170 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721097 2543 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:27:39.728170 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721100 2543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:27:39.728170 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721102 2543 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:27:39.728170 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721105 2543 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:27:39.728170 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721107 2543 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:27:39.728170 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721110 2543 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:27:39.728170 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721112 2543 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:27:39.728170 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721115 2543 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:27:39.728170 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721118 2543 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:27:39.728170 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721120 2543 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:27:39.728170 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721122 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:27:39.728170 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721125 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:27:39.728625 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721127 2543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:27:39.728625 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721130 2543 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:27:39.728625 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721132 2543 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:27:39.728625 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721135 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:27:39.728625 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721137 2543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:27:39.728625 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721140 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:27:39.728625 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721143 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:27:39.728625 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721145 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:27:39.728625 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721147 2543 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:27:39.728625 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721154 2543 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:27:39.728625 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721156 2543 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:27:39.728625 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721159 2543 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:27:39.728625 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721161 2543 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:27:39.728625 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721164 2543 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:27:39.728625 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721167 2543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:27:39.728625 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721170 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:27:39.728625 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721172 2543 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:27:39.728625 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721175 2543 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:27:39.728625 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721177 2543 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:27:39.728625 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721180 2543 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:27:39.729235 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721182 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:27:39.729235 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721185 2543 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:27:39.729235 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721187 2543 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:27:39.729235 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721190 2543 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:27:39.729235 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721192 2543 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:27:39.729235 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721197 2543 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:27:39.729235 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721200 2543 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:27:39.729235 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721203 2543 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:27:39.729235 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721206 2543 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:27:39.729235 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721209 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:27:39.729235 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721211 2543 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:27:39.729235 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721214 2543 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:27:39.729235 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721216 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:27:39.729235 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721219 2543 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:27:39.729235 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721221 2543 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:27:39.729235 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721224 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:27:39.729235 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721226 2543 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:27:39.729235 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721229 2543 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:27:39.729235 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721231 2543 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:27:39.729235 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721234 2543 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:27:39.729850 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721236 2543 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:27:39.729850 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721240 2543 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:27:39.729850 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721243 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:27:39.729850 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721245 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:27:39.729850 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721248 2543 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:27:39.729850 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721251 2543 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:27:39.729850 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721253 2543 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:27:39.729850 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.721276 2543 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:27:39.729850 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721304 2543 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:27:39.729850 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721308 2543 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:27:39.729850 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721311 2543 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:27:39.729850 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721314 2543 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:27:39.729850 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721316 2543 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:27:39.729850 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721319 2543 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:27:39.729850 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721322 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:27:39.729850 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721324 2543 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:27:39.730412 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721327 2543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:27:39.730412 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721329 2543 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:27:39.730412 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721332 2543 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:27:39.730412 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721334 2543 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:27:39.730412 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721337 2543 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:27:39.730412 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721340 2543 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:27:39.730412 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721343 2543 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:27:39.730412 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721345 2543 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:27:39.730412 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721348 2543 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:27:39.730412 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721350 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:27:39.730412 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721353 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:27:39.730412 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721355 2543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:27:39.730412 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721358 2543 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:27:39.730412 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721361 2543 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:27:39.730412 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721363 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:27:39.730412 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721366 2543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:27:39.730412 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721369 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:27:39.730412 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721371 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:27:39.730412 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721375 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:27:39.731083 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721378 2543 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:27:39.731083 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721380 2543 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:27:39.731083 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721385 2543 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:27:39.731083 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721387 2543 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:27:39.731083 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721390 2543 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:27:39.731083 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721392 2543 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:27:39.731083 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721395 2543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:27:39.731083 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721398 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:27:39.731083 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721400 2543 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:27:39.731083 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721403 2543 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:27:39.731083 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721405 2543 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:27:39.731083 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721408 2543 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:27:39.731083 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721411 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:27:39.731083 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721413 2543 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:27:39.731083 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721416 2543 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:27:39.731083 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721418 2543 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:27:39.731083 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721421 2543 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:27:39.731083 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721424 2543 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:27:39.731083 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721429 2543 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:27:39.731083 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721432 2543 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:27:39.731557 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721434 2543 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:27:39.731557 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721437 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:27:39.731557 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721440 2543 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:27:39.731557 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721442 2543 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:27:39.731557 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721445 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:27:39.731557 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721447 2543 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:27:39.731557 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721450 2543 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:27:39.731557 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721452 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:27:39.731557 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721455 2543 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:27:39.731557 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721457 2543 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:27:39.731557 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721460 2543 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:27:39.731557 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721464 2543 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:27:39.731557 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721466 2543 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:27:39.731557 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721469 2543 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:27:39.731557 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721473 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:27:39.731557 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721475 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:27:39.731557 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721478 2543 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:27:39.731557 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721481 2543 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:27:39.731557 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721483 2543 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:27:39.731557 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721486 2543 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:27:39.732048 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721488 2543 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:27:39.732048 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721491 2543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:27:39.732048 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721494 2543 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:27:39.732048 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721496 2543 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:27:39.732048 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721500 2543 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:27:39.732048 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721503 2543 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:27:39.732048 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721506 2543 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:27:39.732048 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721509 2543 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:27:39.732048 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721511 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:27:39.732048 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721514 2543 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:27:39.732048 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721517 2543 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:27:39.732048 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721520 2543 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:27:39.732048 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721523 2543 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:27:39.732048 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721525 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:27:39.732048 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721528 2543 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:27:39.732048 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721530 2543 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:27:39.732048 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721533 2543 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:27:39.732048 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721536 2543 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:27:39.732048 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721538 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:27:39.732520 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721541 2543 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:27:39.732520 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721543 2543 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:27:39.732520 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721546 2543 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:27:39.732520 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721548 2543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:27:39.732520 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721552 2543 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:27:39.732520 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721555 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:27:39.732520 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721558 2543 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:27:39.732520 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721565 2543 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:27:39.732520 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721568 2543 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:27:39.732520 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721571 2543 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:27:39.732520 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721573 2543 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:27:39.732520 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721577 2543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:27:39.732520 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721579 2543 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:27:39.732520 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721582 2543 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:27:39.732520 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721584 2543 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:27:39.732520 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721587 2543 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:27:39.732520 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721591 2543 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:27:39.732520 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721593 2543 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:27:39.732520 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721596 2543 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:27:39.732972 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:39.721599 2543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:27:39.732972 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.722423 2543 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:27:39.732972 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.724842 2543 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 21:27:39.732972 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.725945 2543 server.go:1014] "Starting client certificate rotation" Oct 30 21:27:39.732972 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.726061 2543 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:27:39.732972 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.726911 2543 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:27:39.732972 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.730833 2543 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:27:39.733793 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.733776 2543 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:27:39.747036 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.747019 2543 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:27:39.750880 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.750865 2543 log.go:25] "Validated CRI v1 image API" Oct 30 21:27:39.751776 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.751739 2543 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:27:39.755317 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.755293 2543 fs.go:135] Filesystem UUIDs: map[192550a1-7e00-4901-b523-e4a83ba1c9e8:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a24e2d3f-c530-4c95-913f-b50036f8c3dc:/dev/nvme0n1p4] Oct 30 21:27:39.755381 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.755316 2543 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 21:27:39.759875 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.759762 2543 manager.go:217] Machine: {Timestamp:2026-10-30 21:27:39.759267989 +0000 UTC m=+0.348646153 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100269 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2238eae22eb5ba9498d51db988d5b7 SystemUUID:ec2238ea-e22e-b5ba-9498-d51db988d5b7 BootID:20879828-f4b6-4d33-a897-273cc6bda5ac Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c1:09:2e:63 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c1:09:2e:63 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:62:88:c4:38:85 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:27:39.759875 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.759864 2543 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:27:39.760002 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.759942 2543 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:27:39.762523 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.762502 2543 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:27:39.763483 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.763462 2543 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:27:39.763644 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.763487 2543 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-163.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:27:39.763730 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.763648 2543 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:27:39.763730 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.763661 2543 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:27:39.763730 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.763673 2543 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:27:39.763730 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.763690 2543 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:27:39.765016 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.765003 2543 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:27:39.769686 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.769672 2543 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:27:39.769835 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.769824 2543 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:27:39.769893 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.769863 2543 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:27:39.769893 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.769876 2543 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:27:39.769893 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.769888 2543 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:27:39.770014 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.769901 2543 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:27:39.770014 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.769914 2543 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:27:39.770250 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.770236 2543 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:27:39.773666 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.773650 2543 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:27:39.780559 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.780543 2543 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:27:39.781550 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.781538 2543 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:27:39.781595 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.781568 2543 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:27:39.781731 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.781721 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:27:39.781774 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.781737 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:27:39.781774 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.781743 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:27:39.781774 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.781762 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:27:39.781774 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.781771 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:27:39.781872 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.781777 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:27:39.782432 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.782423 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:27:39.782462 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.782432 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:27:39.782462 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.782439 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:27:39.782462 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.782445 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:27:39.782462 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.782454 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:27:39.782462 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.782464 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:27:39.783420 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.783404 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:27:39.783420 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.783419 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:27:39.784192 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.784175 2543 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hvjdz" Oct 30 21:27:39.786300 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.786259 2543 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-163.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 21:27:39.787213 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.787201 2543 server.go:1263] "Started kubelet" Oct 30 21:27:39.787329 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.787270 2543 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:27:39.787383 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.787282 2543 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:27:39.787383 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.787345 2543 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:27:39.787988 ip-10-0-140-163 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:27:39.788929 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.788911 2543 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:27:39.789186 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.789174 2543 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:27:39.795391 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:39.795363 2543 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 21:27:39.797059 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.797046 2543 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:27:39.797622 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.797600 2543 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:27:39.797683 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:39.797673 2543 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 21:27:39.797740 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.797691 2543 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hvjdz" Oct 30 21:27:39.798297 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.798280 2543 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:27:39.798297 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.798298 2543 factory.go:55] Registering systemd factory Oct 30 21:27:39.798413 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.798308 2543 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:27:39.798413 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.798386 2543 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:27:39.798413 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.798400 2543 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:27:39.798413 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.798391 2543 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:27:39.798546 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.798481 2543 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:27:39.798546 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.798488 2543 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:27:39.798616 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.798591 2543 factory.go:153] Registering CRI-O factory Oct 30 21:27:39.798616 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.798602 2543 factory.go:223] Registration of the crio container factory successfully Oct 30 21:27:39.798695 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.798622 2543 factory.go:103] Registering Raw factory Oct 30 21:27:39.798695 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.798638 2543 manager.go:1199] Started watching for new ooms in manager Oct 30 21:27:39.800224 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:39.800201 2543 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-163.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 21:27:39.800224 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:39.800202 2543 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 21:27:39.800380 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.800286 2543 manager.go:319] Starting recovery of all containers Oct 30 21:27:39.801446 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:39.800281 2543 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-163.us-east-2.compute.internal.18e36bd9cb5781e2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-163.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-163.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 21:27:39.78717437 +0000 UTC m=+0.376552532,LastTimestamp:2026-10-30 21:27:39.78717437 +0000 UTC m=+0.376552532,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-163.us-east-2.compute.internal,}" Oct 30 21:27:39.807404 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.807384 2543 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:27:39.811671 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.811652 2543 manager.go:324] Recovery completed Oct 30 21:27:39.811930 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:39.811891 2543 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-163.us-east-2.compute.internal\" not found" node="ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:27:39.815669 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.815655 2543 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:27:39.817288 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.817271 2543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:27:39.817371 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.817303 2543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:27:39.817371 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.817319 2543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:27:39.817896 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.817879 2543 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:27:39.817896 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.817892 2543 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:27:39.817996 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.817892 2543 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:27:39.817996 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.817931 2543 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:27:39.818531 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.818517 2543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:27:39.818579 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.818541 2543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:27:39.818579 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.818551 2543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:27:39.965306 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:39.898081 2543 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 21:27:39.965306 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.964043 2543 policy_none.go:50] "Start" Oct 30 21:27:39.965306 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.964063 2543 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:27:39.965306 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.964079 2543 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:27:39.967308 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.966023 2543 policy_none.go:44] "Start" Oct 30 21:27:39.979811 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.979772 2543 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:27:39.981114 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.981098 2543 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:27:39.981183 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.981120 2543 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:27:39.981183 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.981137 2543 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:27:39.981183 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:39.981167 2543 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:27:39.985316 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:39.985295 2543 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:27:39.999137 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:39.999119 2543 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 21:27:40.004292 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.004271 2543 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:27:40.006489 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.006477 2543 manager.go:341] "Starting Device Plugin manager" Oct 30 21:27:40.007300 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:40.007186 2543 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:27:40.007351 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.007301 2543 server.go:87] "Starting device plugin registration server" Oct 30 21:27:40.007502 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.007490 2543 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:27:40.007535 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.007505 2543 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:27:40.007611 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.007588 2543 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:27:40.007685 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.007665 2543 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:27:40.007685 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.007673 2543 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:27:40.008242 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:40.008222 2543 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:27:40.008312 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:40.008277 2543 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 21:27:40.081479 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.081457 2543 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal"] Oct 30 21:27:40.081568 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.081508 2543 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:27:40.082317 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.082301 2543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:27:40.082418 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.082332 2543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:27:40.082418 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.082348 2543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:27:40.084693 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.084678 2543 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:27:40.084857 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.084843 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:27:40.084908 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.084871 2543 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:27:40.085400 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.085385 2543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:27:40.085400 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.085394 2543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:27:40.085484 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.085411 2543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:27:40.085484 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.085419 2543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:27:40.085484 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.085425 2543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:27:40.085484 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.085433 2543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:27:40.087524 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.087507 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:27:40.087580 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.087530 2543 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:27:40.088198 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.088183 2543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:27:40.088280 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.088210 2543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:27:40.088280 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.088221 2543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:27:40.100677 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.100659 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40b5c76fae82f6d7a38a1a745d5d3947-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"40b5c76fae82f6d7a38a1a745d5d3947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:27:40.100744 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.100686 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40b5c76fae82f6d7a38a1a745d5d3947-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"40b5c76fae82f6d7a38a1a745d5d3947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:27:40.100744 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.100702 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b404058373133d538844156a64ac398-config\") pod \"kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"3b404058373133d538844156a64ac398\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:27:40.108262 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.108250 2543 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:27:40.109044 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.109031 2543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:27:40.109111 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.109059 2543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:27:40.109111 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.109069 2543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:27:40.109111 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.109089 2543 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:27:40.116198 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:40.116182 2543 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" node="ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:27:40.118164 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:40.118143 2543 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" node="ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:27:40.124798 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.124786 2543 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:27:40.124846 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:40.124804 2543 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-163.us-east-2.compute.internal\": node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 21:27:40.144876 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:40.144857 2543 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 21:27:40.201733 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.201708 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40b5c76fae82f6d7a38a1a745d5d3947-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"40b5c76fae82f6d7a38a1a745d5d3947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:27:40.201841 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.201740 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40b5c76fae82f6d7a38a1a745d5d3947-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"40b5c76fae82f6d7a38a1a745d5d3947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:27:40.201841 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.201779 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b404058373133d538844156a64ac398-config\") pod \"kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"3b404058373133d538844156a64ac398\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:27:40.201841 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.201823 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40b5c76fae82f6d7a38a1a745d5d3947-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"40b5c76fae82f6d7a38a1a745d5d3947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:27:40.201947 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.201824 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b404058373133d538844156a64ac398-config\") pod \"kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"3b404058373133d538844156a64ac398\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:27:40.201947 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.201823 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40b5c76fae82f6d7a38a1a745d5d3947-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"40b5c76fae82f6d7a38a1a745d5d3947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:27:40.245531 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:40.245487 2543 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 21:27:40.345761 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:40.345727 2543 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 21:27:40.421169 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.421149 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:27:40.423318 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.423302 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:27:40.446078 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:40.446046 2543 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 21:27:40.546619 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:40.546554 2543 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 21:27:40.646983 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:40.646959 2543 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 21:27:40.725601 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.725582 2543 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 21:27:40.726038 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.725690 2543 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:27:40.726038 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.725729 2543 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:27:40.747878 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:40.747856 2543 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 21:27:40.803386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.803316 2543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:22:39 +0000 UTC" deadline="2028-04-06 02:27:51.950872879 +0000 UTC" Oct 30 21:27:40.803386 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:40.803352 2543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12557h0m11.147523383s" Oct 30 21:27:40.848152 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:40.848134 2543 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 21:27:40.948599 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:40.948573 2543 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 21:27:41.049349 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:41.049314 2543 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 21:27:41.050230 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:41.050196 2543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b5c76fae82f6d7a38a1a745d5d3947.slice/crio-a23d62cf087a22dbe4e7bf329a60a92adba8cb51a46683861e9baba4d8d3c18b WatchSource:0}: Error finding container a23d62cf087a22dbe4e7bf329a60a92adba8cb51a46683861e9baba4d8d3c18b: Status 404 returned error can't find the container with id a23d62cf087a22dbe4e7bf329a60a92adba8cb51a46683861e9baba4d8d3c18b Oct 30 21:27:41.050636 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:41.050616 2543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b404058373133d538844156a64ac398.slice/crio-edbef9c655815b9877f936c73f1f75f3449440552d90723d5d0a8f2b9727a8da WatchSource:0}: Error finding container edbef9c655815b9877f936c73f1f75f3449440552d90723d5d0a8f2b9727a8da: Status 404 returned error can't find the container with id edbef9c655815b9877f936c73f1f75f3449440552d90723d5d0a8f2b9727a8da Oct 30 21:27:41.055893 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.055878 2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:27:41.131165 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.131150 2543 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:27:41.197886 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.197870 2543 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:27:41.210112 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.210091 2543 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:27:41.212009 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.211995 2543 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:27:41.219191 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.219177 2543 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:27:41.406041 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.405991 2543 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:27:41.771482 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.771425 2543 apiserver.go:52] "Watching apiserver" Oct 30 21:27:41.783200 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.783175 2543 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:27:41.783476 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.783459 2543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-kv4cs","openshift-ovn-kubernetes/ovnkube-node-xh6hf","kube-system/konnectivity-agent-7d6nt","kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-6fx47","openshift-image-registry/node-ca-db7tz","openshift-multus/multus-additional-cni-plugins-vcn8r","openshift-multus/multus-hl8qs","openshift-dns/node-resolver-j2j9f","openshift-multus/network-metrics-daemon-4r4zb","openshift-network-operator/iptables-alerter-wd8lr","kube-system/global-pull-secret-syncer-c2vrg"] Oct 30 21:27:41.785727 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.785706 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.787849 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.787830 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:27:41.789782 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.789741 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:27:41.789877 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.789864 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:27:41.790026 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.790010 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-99k6q\"" Oct 30 21:27:41.790095 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.790084 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:27:41.790160 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:41.790138 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kv4cs" podUID="bb88c1ff-4b28-4812-9b2b-8876983720e5" Oct 30 21:27:41.791722 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.791702 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:27:41.791871 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.791804 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:27:41.791926 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.791893 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:27:41.792005 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.791991 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:27:41.792050 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.792036 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:27:41.792102 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.792050 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-flwfj\"" Oct 30 21:27:41.792388 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.792370 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.795630 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.795465 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j2j9f" Oct 30 21:27:41.797771 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.797562 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.799331 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.799316 2543 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 21:27:41.799847 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.799823 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7d6nt" Oct 30 21:27:41.803558 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.802028 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wd8lr" Oct 30 21:27:41.804410 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.804002 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:27:41.806768 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.806203 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:27:41.806768 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:41.806265 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zb" podUID="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" Oct 30 21:27:41.808540 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.808520 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-var-lib-kubelet\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.808621 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.808554 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:27:41.808621 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.808582 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-run\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.808621 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.808603 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-slash\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.808789 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.808625 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-multus-cni-dir\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.808789 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.808648 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-run-netns\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.808789 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.808687 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-var-lib-cni-bin\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.808789 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.808710 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/950f0225-e1c9-4c9c-8fc6-25dba5db7120-agent-certs\") pod \"konnectivity-agent-7d6nt\" (UID: \"950f0225-e1c9-4c9c-8fc6-25dba5db7120\") " pod="kube-system/konnectivity-agent-7d6nt" Oct 30 21:27:41.808789 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.808731 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-modprobe-d\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.809029 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.808798 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-lib-modules\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.809029 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.808850 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-run-systemd\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.809029 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.808876 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-system-cni-dir\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:27:41.809029 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.808908 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:27:41.809029 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.808933 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dpq2\" (UniqueName: \"kubernetes.io/projected/6841b7e6-42ec-493f-9613-ac4902212fac-kube-api-access-2dpq2\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.809029 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.808956 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-multus-daemon-config\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.809029 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.808978 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-run-multus-certs\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.809334 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809035 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqcwl\" (UniqueName: \"kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl\") pod \"network-check-target-kv4cs\" (UID: \"bb88c1ff-4b28-4812-9b2b-8876983720e5\") " pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:27:41.809334 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809059 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-sysconfig\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.809334 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809081 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-systemd\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.809334 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809103 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-tuned\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.809334 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809138 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-run-netns\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.809334 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809161 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-run-ovn\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.809334 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809181 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-cnibin\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.809334 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809204 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-multus-socket-dir-parent\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.809334 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809245 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3af705e-b3ba-4379-8214-07ba1b431824-tmp-dir\") pod \"node-resolver-j2j9f\" (UID: \"c3af705e-b3ba-4379-8214-07ba1b431824\") " pod="openshift-dns/node-resolver-j2j9f" Oct 30 21:27:41.809334 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809269 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.809334 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809308 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-db7tz" Oct 30 21:27:41.809334 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809323 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-etc-openvswitch\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.809926 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809347 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-node-log\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.809926 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809388 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-log-socket\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.809926 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809418 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:27:41.809926 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809447 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-run-ovn-kubernetes\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.809926 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809469 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6841b7e6-42ec-493f-9613-ac4902212fac-ovnkube-config\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.809926 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809521 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:27:41.809926 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809547 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c3af705e-b3ba-4379-8214-07ba1b431824-hosts-file\") pod \"node-resolver-j2j9f\" (UID: \"c3af705e-b3ba-4379-8214-07ba1b431824\") " pod="openshift-dns/node-resolver-j2j9f" Oct 30 21:27:41.809926 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809577 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sm8p\" (UniqueName: \"kubernetes.io/projected/c3af705e-b3ba-4379-8214-07ba1b431824-kube-api-access-8sm8p\") pod \"node-resolver-j2j9f\" (UID: \"c3af705e-b3ba-4379-8214-07ba1b431824\") " pod="openshift-dns/node-resolver-j2j9f" Oct 30 21:27:41.809926 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809599 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/950f0225-e1c9-4c9c-8fc6-25dba5db7120-konnectivity-ca\") pod \"konnectivity-agent-7d6nt\" (UID: \"950f0225-e1c9-4c9c-8fc6-25dba5db7120\") " pod="kube-system/konnectivity-agent-7d6nt" Oct 30 21:27:41.809926 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809634 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-var-lib-cni-multus\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.809926 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809659 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-sysctl-conf\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.809926 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809695 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-host\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.809926 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809791 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-cnibin\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:27:41.809926 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809823 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-run-openvswitch\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.809926 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809845 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-cni-netd\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.809926 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809869 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6841b7e6-42ec-493f-9613-ac4902212fac-ovnkube-script-lib\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.810719 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809891 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-var-lib-kubelet\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.810719 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809912 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-sysctl-d\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.810719 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809938 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-kubelet\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.810719 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.809973 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-var-lib-openvswitch\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.810719 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.810030 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6841b7e6-42ec-493f-9613-ac4902212fac-ovn-node-metrics-cert\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.810719 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.810062 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-multus-conf-dir\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.810719 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.810076 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krfth\" (UniqueName: \"kubernetes.io/projected/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-kube-api-access-krfth\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.810719 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.810108 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-systemd-units\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.810719 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.810123 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-system-cni-dir\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.810719 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.810141 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-etc-kubernetes\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.810719 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.810165 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-run-k8s-cni-cncf-io\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.810719 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.810238 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hgqm\" (UniqueName: \"kubernetes.io/projected/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-kube-api-access-2hgqm\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:27:41.810719 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.810261 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-cni-binary-copy\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.810719 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.810279 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-tmp\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.810719 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.810307 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d92bb\" (UniqueName: \"kubernetes.io/projected/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-kube-api-access-d92bb\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.810719 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.810330 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-cni-bin\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.810719 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.810348 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6841b7e6-42ec-493f-9613-ac4902212fac-env-overrides\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.811433 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.810364 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-hostroot\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.811433 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.810409 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-kubernetes\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.811433 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.810423 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-sys\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.811433 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.811123 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:27:41.811433 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:41.811185 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c2vrg" podUID="183318eb-952e-4e93-b716-6ab50b9c58dd" Oct 30 21:27:41.826048 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.826032 2543 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:27:41.881789 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.881767 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:27:41.881789 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.881778 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:27:41.881933 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.881913 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:27:41.882029 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.882016 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:27:41.882068 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.882059 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:27:41.882133 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.882124 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:27:41.882201 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.882191 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:27:41.882237 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.882226 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:27:41.882407 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.882362 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:27:41.882445 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.882426 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:27:41.882445 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.882440 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:27:41.883336 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.883313 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:27:41.883777 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.883745 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:27:41.883861 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.883795 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:27:41.883861 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.883835 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:27:41.883940 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.883884 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:27:41.883940 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.883919 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:27:41.884741 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.884726 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:27:41.884837 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.884765 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pqqf8\"" Oct 30 21:27:41.884837 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.884800 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2pcx6\"" Oct 30 21:27:41.884937 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.884838 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:27:41.884937 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.884898 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g2c5m\"" Oct 30 21:27:41.884937 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.884908 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-z8x7v\"" Oct 30 21:27:41.885073 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.884938 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d996q\"" Oct 30 21:27:41.885073 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.884841 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wxqgz\"" Oct 30 21:27:41.885073 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.884992 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ct8zc\"" Oct 30 21:27:41.885073 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.885016 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:27:41.899794 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.899774 2543 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:27:41.911786 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.910734 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6841b7e6-42ec-493f-9613-ac4902212fac-ovnkube-config\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.911786 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.910791 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:27:41.911786 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.910815 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c3af705e-b3ba-4379-8214-07ba1b431824-hosts-file\") pod \"node-resolver-j2j9f\" (UID: \"c3af705e-b3ba-4379-8214-07ba1b431824\") " pod="openshift-dns/node-resolver-j2j9f" Oct 30 21:27:41.911786 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.910837 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sm8p\" (UniqueName: \"kubernetes.io/projected/c3af705e-b3ba-4379-8214-07ba1b431824-kube-api-access-8sm8p\") pod \"node-resolver-j2j9f\" (UID: \"c3af705e-b3ba-4379-8214-07ba1b431824\") " pod="openshift-dns/node-resolver-j2j9f" Oct 30 21:27:41.911786 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.910860 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/950f0225-e1c9-4c9c-8fc6-25dba5db7120-konnectivity-ca\") pod \"konnectivity-agent-7d6nt\" (UID: \"950f0225-e1c9-4c9c-8fc6-25dba5db7120\") " pod="kube-system/konnectivity-agent-7d6nt" Oct 30 21:27:41.911786 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.910881 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-var-lib-cni-multus\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.911786 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.910902 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-sysctl-conf\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.911786 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.910920 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-host\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.911786 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.910924 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c3af705e-b3ba-4379-8214-07ba1b431824-hosts-file\") pod \"node-resolver-j2j9f\" (UID: \"c3af705e-b3ba-4379-8214-07ba1b431824\") " pod="openshift-dns/node-resolver-j2j9f" Oct 30 21:27:41.911786 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.910940 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-cnibin\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:27:41.911786 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.910959 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-run-openvswitch\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.911786 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.910981 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-var-lib-cni-multus\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.911786 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.910994 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-cni-netd\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.911786 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911014 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6841b7e6-42ec-493f-9613-ac4902212fac-ovnkube-script-lib\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.911786 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911034 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-var-lib-kubelet\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.911786 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911057 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-sysctl-d\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.911786 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911078 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-kubelet\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.912449 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911102 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-var-lib-openvswitch\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.912449 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911108 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-run-openvswitch\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.912449 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911122 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6841b7e6-42ec-493f-9613-ac4902212fac-ovn-node-metrics-cert\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.912449 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911143 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-multus-conf-dir\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.912449 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911160 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krfth\" (UniqueName: \"kubernetes.io/projected/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-kube-api-access-krfth\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.912449 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911180 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:27:41.912449 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911199 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx5l7\" (UniqueName: \"kubernetes.io/projected/5934c36a-2955-4f0c-a8e8-615f2fa07454-kube-api-access-mx5l7\") pod \"node-ca-db7tz\" (UID: \"5934c36a-2955-4f0c-a8e8-615f2fa07454\") " pod="openshift-image-registry/node-ca-db7tz" Oct 30 21:27:41.912449 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911222 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-systemd-units\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.912449 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911235 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-sysctl-conf\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.912449 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911243 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-system-cni-dir\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.912449 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911283 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-host\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.912449 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911293 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-system-cni-dir\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.912449 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911307 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:27:41.912449 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911311 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6841b7e6-42ec-493f-9613-ac4902212fac-ovnkube-config\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.912449 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911306 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-etc-kubernetes\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.912449 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911333 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-cni-netd\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.912449 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911343 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-cnibin\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:27:41.913129 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911354 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5934c36a-2955-4f0c-a8e8-615f2fa07454-serviceca\") pod \"node-ca-db7tz\" (UID: \"5934c36a-2955-4f0c-a8e8-615f2fa07454\") " pod="openshift-image-registry/node-ca-db7tz" Oct 30 21:27:41.913129 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911367 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-etc-kubernetes\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.913129 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911377 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ae36e61-388d-4bc0-8810-aae71f3703c6-host-slash\") pod \"iptables-alerter-wd8lr\" (UID: \"2ae36e61-388d-4bc0-8810-aae71f3703c6\") " pod="openshift-network-operator/iptables-alerter-wd8lr" Oct 30 21:27:41.913129 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911386 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-var-lib-kubelet\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.913129 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911396 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-run-k8s-cni-cncf-io\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.913129 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911403 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-multus-conf-dir\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.913129 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911422 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-run-k8s-cni-cncf-io\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.913129 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911424 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-registration-dir\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:27:41.913129 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911429 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/950f0225-e1c9-4c9c-8fc6-25dba5db7120-konnectivity-ca\") pod \"konnectivity-agent-7d6nt\" (UID: \"950f0225-e1c9-4c9c-8fc6-25dba5db7120\") " pod="kube-system/konnectivity-agent-7d6nt" Oct 30 21:27:41.913129 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911475 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-systemd-units\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.913129 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911481 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-kubelet\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.913129 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911485 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-var-lib-openvswitch\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.913129 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911505 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2ae36e61-388d-4bc0-8810-aae71f3703c6-iptables-alerter-script\") pod \"iptables-alerter-wd8lr\" (UID: \"2ae36e61-388d-4bc0-8810-aae71f3703c6\") " pod="openshift-network-operator/iptables-alerter-wd8lr" Oct 30 21:27:41.913129 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911541 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-sysctl-d\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.913129 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911556 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hgqm\" (UniqueName: \"kubernetes.io/projected/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-kube-api-access-2hgqm\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:27:41.913129 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911582 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-cni-binary-copy\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.913129 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911655 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-device-dir\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:27:41.913857 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911767 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:27:41.913857 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911771 2543 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:27:41.913857 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911800 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs\") pod \"network-metrics-daemon-4r4zb\" (UID: \"3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78\") " pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:27:41.913857 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911811 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6841b7e6-42ec-493f-9613-ac4902212fac-ovnkube-script-lib\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.913857 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911824 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gstg\" (UniqueName: \"kubernetes.io/projected/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-kube-api-access-4gstg\") pod \"network-metrics-daemon-4r4zb\" (UID: \"3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78\") " pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:27:41.913857 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911846 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-tmp\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.913857 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911872 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d92bb\" (UniqueName: \"kubernetes.io/projected/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-kube-api-access-d92bb\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.913857 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911938 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-cni-bin\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.913857 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911954 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6841b7e6-42ec-493f-9613-ac4902212fac-env-overrides\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.913857 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.911995 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-hostroot\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.913857 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912030 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-socket-dir\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:27:41.913857 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912057 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5934c36a-2955-4f0c-a8e8-615f2fa07454-host\") pod \"node-ca-db7tz\" (UID: \"5934c36a-2955-4f0c-a8e8-615f2fa07454\") " pod="openshift-image-registry/node-ca-db7tz" Oct 30 21:27:41.913857 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912060 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-cni-binary-copy\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.913857 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912124 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-kubernetes\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.913857 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912166 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-sys\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.913857 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912189 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-kubernetes\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.913857 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912204 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-var-lib-kubelet\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.913857 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912228 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-hostroot\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.914590 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912230 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:27:41.914590 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912277 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjtk6\" (UniqueName: \"kubernetes.io/projected/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-kube-api-access-qjtk6\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:27:41.914590 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912299 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-cni-bin\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.914590 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912304 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret\") pod \"global-pull-secret-syncer-c2vrg\" (UID: \"183318eb-952e-4e93-b716-6ab50b9c58dd\") " pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:27:41.914590 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912280 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-sys\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.914590 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912331 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6841b7e6-42ec-493f-9613-ac4902212fac-env-overrides\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.914590 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912340 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-run\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.914590 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912357 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-var-lib-kubelet\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.914590 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912387 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-slash\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.914590 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912428 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-multus-cni-dir\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.914590 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912442 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-slash\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.914590 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912461 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-run\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.914590 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912465 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-run-netns\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.914590 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912507 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-run-netns\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.914590 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912532 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-var-lib-cni-bin\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.914590 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912552 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/950f0225-e1c9-4c9c-8fc6-25dba5db7120-agent-certs\") pod \"konnectivity-agent-7d6nt\" (UID: \"950f0225-e1c9-4c9c-8fc6-25dba5db7120\") " pod="kube-system/konnectivity-agent-7d6nt" Oct 30 21:27:41.914590 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912551 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-multus-cni-dir\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.914590 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912577 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-var-lib-cni-bin\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.915360 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912616 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-modprobe-d\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.915360 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912635 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-lib-modules\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.915360 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912644 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:27:41.915360 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912653 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-run-systemd\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.915360 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912683 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-sys-fs\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:27:41.915360 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912703 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-system-cni-dir\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:27:41.915360 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912735 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-modprobe-d\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.915360 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912746 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:27:41.915360 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912784 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dpq2\" (UniqueName: \"kubernetes.io/projected/6841b7e6-42ec-493f-9613-ac4902212fac-kube-api-access-2dpq2\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.915360 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912787 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-run-systemd\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.915360 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912809 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-multus-daemon-config\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.915360 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912818 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-system-cni-dir\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:27:41.915360 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912838 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-run-multus-certs\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.915360 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912887 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqcwl\" (UniqueName: \"kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl\") pod \"network-check-target-kv4cs\" (UID: \"bb88c1ff-4b28-4812-9b2b-8876983720e5\") " pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:27:41.915360 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912914 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-sysconfig\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.915360 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912921 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-lib-modules\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.915360 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912934 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-systemd\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.915968 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912957 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-run-multus-certs\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.915968 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.912958 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-tuned\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.915968 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913020 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-run-netns\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.915968 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913044 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-run-ovn\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.915968 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913063 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-cnibin\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.915968 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913085 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-multus-socket-dir-parent\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.915968 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913114 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:27:41.915968 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913119 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3af705e-b3ba-4379-8214-07ba1b431824-tmp-dir\") pod \"node-resolver-j2j9f\" (UID: \"c3af705e-b3ba-4379-8214-07ba1b431824\") " pod="openshift-dns/node-resolver-j2j9f" Oct 30 21:27:41.915968 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913326 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3af705e-b3ba-4379-8214-07ba1b431824-tmp-dir\") pod \"node-resolver-j2j9f\" (UID: \"c3af705e-b3ba-4379-8214-07ba1b431824\") " pod="openshift-dns/node-resolver-j2j9f" Oct 30 21:27:41.915968 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913362 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-cnibin\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.915968 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913383 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-run-netns\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.915968 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913411 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-systemd\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.915968 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913380 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-sysconfig\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.915968 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913410 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.915968 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913441 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.915968 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913447 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-multus-socket-dir-parent\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.915968 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913480 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwp8n\" (UniqueName: \"kubernetes.io/projected/2ae36e61-388d-4bc0-8810-aae71f3703c6-kube-api-access-vwp8n\") pod \"iptables-alerter-wd8lr\" (UID: \"2ae36e61-388d-4bc0-8810-aae71f3703c6\") " pod="openshift-network-operator/iptables-alerter-wd8lr" Oct 30 21:27:41.915968 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913491 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-run-ovn\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.916708 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913530 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-etc-openvswitch\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.916708 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913569 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-etc-openvswitch\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.916708 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913568 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-node-log\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.916708 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913599 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-log-socket\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.916708 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913603 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-node-log\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.916708 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913628 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/183318eb-952e-4e93-b716-6ab50b9c58dd-kubelet-config\") pod \"global-pull-secret-syncer-c2vrg\" (UID: \"183318eb-952e-4e93-b716-6ab50b9c58dd\") " pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:27:41.916708 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913639 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-log-socket\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.916708 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913681 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/183318eb-952e-4e93-b716-6ab50b9c58dd-dbus\") pod \"global-pull-secret-syncer-c2vrg\" (UID: \"183318eb-952e-4e93-b716-6ab50b9c58dd\") " pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:27:41.916708 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913705 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:27:41.916708 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913777 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-run-ovn-kubernetes\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.916708 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.913845 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-run-ovn-kubernetes\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.916708 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.914168 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-multus-daemon-config\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.916708 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.914334 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:27:41.916708 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.915496 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-tmp\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.916708 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.915538 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-tuned\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.916708 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.915986 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/950f0225-e1c9-4c9c-8fc6-25dba5db7120-agent-certs\") pod \"konnectivity-agent-7d6nt\" (UID: \"950f0225-e1c9-4c9c-8fc6-25dba5db7120\") " pod="kube-system/konnectivity-agent-7d6nt" Oct 30 21:27:41.917379 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.917055 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6841b7e6-42ec-493f-9613-ac4902212fac-ovn-node-metrics-cert\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.922500 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.922472 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hgqm\" (UniqueName: \"kubernetes.io/projected/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-kube-api-access-2hgqm\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:27:41.924508 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.924491 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krfth\" (UniqueName: \"kubernetes.io/projected/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-kube-api-access-krfth\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:27:41.924779 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.924747 2543 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:27:41.924862 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.924845 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sm8p\" (UniqueName: \"kubernetes.io/projected/c3af705e-b3ba-4379-8214-07ba1b431824-kube-api-access-8sm8p\") pod \"node-resolver-j2j9f\" (UID: \"c3af705e-b3ba-4379-8214-07ba1b431824\") " pod="openshift-dns/node-resolver-j2j9f" Oct 30 21:27:41.925389 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:41.925375 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:27:41.925426 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:41.925396 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:27:41.925426 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:41.925407 2543 projected.go:196] Error preparing data for projected volume kube-api-access-qqcwl for pod openshift-network-diagnostics/network-check-target-kv4cs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:27:41.925482 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:41.925454 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl podName:bb88c1ff-4b28-4812-9b2b-8876983720e5 nodeName:}" failed. No retries permitted until 2026-10-30 21:27:42.425436728 +0000 UTC m=+3.014814876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qqcwl" (UniqueName: "kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl") pod "network-check-target-kv4cs" (UID: "bb88c1ff-4b28-4812-9b2b-8876983720e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:27:41.929332 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.929218 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d92bb\" (UniqueName: \"kubernetes.io/projected/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-kube-api-access-d92bb\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:41.930128 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.930111 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dpq2\" (UniqueName: \"kubernetes.io/projected/6841b7e6-42ec-493f-9613-ac4902212fac-kube-api-access-2dpq2\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:41.938059 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.937350 2543 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bwjp5" Oct 30 21:27:41.940653 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.940638 2543 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:27:41.945588 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.945573 2543 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bwjp5" Oct 30 21:27:41.986102 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.986056 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" event={"ID":"40b5c76fae82f6d7a38a1a745d5d3947","Type":"ContainerStarted","Data":"a23d62cf087a22dbe4e7bf329a60a92adba8cb51a46683861e9baba4d8d3c18b"} Oct 30 21:27:41.987186 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:41.987158 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" event={"ID":"3b404058373133d538844156a64ac398","Type":"ContainerStarted","Data":"edbef9c655815b9877f936c73f1f75f3449440552d90723d5d0a8f2b9727a8da"} Oct 30 21:27:42.014500 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.014468 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/183318eb-952e-4e93-b716-6ab50b9c58dd-kubelet-config\") pod \"global-pull-secret-syncer-c2vrg\" (UID: \"183318eb-952e-4e93-b716-6ab50b9c58dd\") " pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:27:42.014596 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.014539 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/183318eb-952e-4e93-b716-6ab50b9c58dd-kubelet-config\") pod \"global-pull-secret-syncer-c2vrg\" (UID: \"183318eb-952e-4e93-b716-6ab50b9c58dd\") " pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:27:42.014596 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.014572 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/183318eb-952e-4e93-b716-6ab50b9c58dd-dbus\") pod \"global-pull-secret-syncer-c2vrg\" (UID: \"183318eb-952e-4e93-b716-6ab50b9c58dd\") " pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:27:42.014773 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.014611 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:27:42.014773 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.014634 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx5l7\" (UniqueName: \"kubernetes.io/projected/5934c36a-2955-4f0c-a8e8-615f2fa07454-kube-api-access-mx5l7\") pod \"node-ca-db7tz\" (UID: \"5934c36a-2955-4f0c-a8e8-615f2fa07454\") " pod="openshift-image-registry/node-ca-db7tz" Oct 30 21:27:42.014773 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.014661 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5934c36a-2955-4f0c-a8e8-615f2fa07454-serviceca\") pod \"node-ca-db7tz\" (UID: \"5934c36a-2955-4f0c-a8e8-615f2fa07454\") " pod="openshift-image-registry/node-ca-db7tz" Oct 30 21:27:42.014773 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.014682 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ae36e61-388d-4bc0-8810-aae71f3703c6-host-slash\") pod \"iptables-alerter-wd8lr\" (UID: \"2ae36e61-388d-4bc0-8810-aae71f3703c6\") " pod="openshift-network-operator/iptables-alerter-wd8lr" Oct 30 21:27:42.014773 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.014696 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/183318eb-952e-4e93-b716-6ab50b9c58dd-dbus\") pod \"global-pull-secret-syncer-c2vrg\" (UID: \"183318eb-952e-4e93-b716-6ab50b9c58dd\") " pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:27:42.014773 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.014704 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-registration-dir\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:27:42.014773 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.014738 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2ae36e61-388d-4bc0-8810-aae71f3703c6-iptables-alerter-script\") pod \"iptables-alerter-wd8lr\" (UID: \"2ae36e61-388d-4bc0-8810-aae71f3703c6\") " pod="openshift-network-operator/iptables-alerter-wd8lr" Oct 30 21:27:42.014773 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.014766 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ae36e61-388d-4bc0-8810-aae71f3703c6-host-slash\") pod \"iptables-alerter-wd8lr\" (UID: \"2ae36e61-388d-4bc0-8810-aae71f3703c6\") " pod="openshift-network-operator/iptables-alerter-wd8lr" Oct 30 21:27:42.015092 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.014766 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:27:42.015092 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.014786 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-device-dir\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:27:42.015092 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.014810 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:27:42.015092 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.014833 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs\") pod \"network-metrics-daemon-4r4zb\" (UID: \"3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78\") " pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:27:42.015092 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.014860 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gstg\" (UniqueName: \"kubernetes.io/projected/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-kube-api-access-4gstg\") pod \"network-metrics-daemon-4r4zb\" (UID: \"3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78\") " pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:27:42.015092 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.014864 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-device-dir\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:27:42.015092 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.014869 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-registration-dir\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:27:42.015092 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.014884 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-socket-dir\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:27:42.015092 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.014907 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5934c36a-2955-4f0c-a8e8-615f2fa07454-host\") pod \"node-ca-db7tz\" (UID: \"5934c36a-2955-4f0c-a8e8-615f2fa07454\") " pod="openshift-image-registry/node-ca-db7tz" Oct 30 21:27:42.015092 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.014933 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjtk6\" (UniqueName: \"kubernetes.io/projected/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-kube-api-access-qjtk6\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:27:42.025935 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:42.025889 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:27:42.026022 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:42.026003 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret podName:183318eb-952e-4e93-b716-6ab50b9c58dd nodeName:}" failed. No retries permitted until 2026-10-30 21:27:42.525989168 +0000 UTC m=+3.115367328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret") pod "global-pull-secret-syncer-c2vrg" (UID: "183318eb-952e-4e93-b716-6ab50b9c58dd") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:27:42.026242 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.026226 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5934c36a-2955-4f0c-a8e8-615f2fa07454-host\") pod \"node-ca-db7tz\" (UID: \"5934c36a-2955-4f0c-a8e8-615f2fa07454\") " pod="openshift-image-registry/node-ca-db7tz" Oct 30 21:27:42.026677 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.026453 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-socket-dir\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:27:42.026677 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.026459 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret\") pod \"global-pull-secret-syncer-c2vrg\" (UID: \"183318eb-952e-4e93-b716-6ab50b9c58dd\") " pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:27:42.026677 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.026504 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:27:42.026677 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:42.014941 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:27:42.026677 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.026542 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-sys-fs\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:27:42.026677 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:42.026603 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs podName:3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78 nodeName:}" failed. No retries permitted until 2026-10-30 21:27:42.526585611 +0000 UTC m=+3.115963775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs") pod "network-metrics-daemon-4r4zb" (UID: "3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:27:42.026677 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.026628 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwp8n\" (UniqueName: \"kubernetes.io/projected/2ae36e61-388d-4bc0-8810-aae71f3703c6-kube-api-access-vwp8n\") pod \"iptables-alerter-wd8lr\" (UID: \"2ae36e61-388d-4bc0-8810-aae71f3703c6\") " pod="openshift-network-operator/iptables-alerter-wd8lr" Oct 30 21:27:42.026677 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.026638 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-sys-fs\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:27:42.026677 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.026642 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5934c36a-2955-4f0c-a8e8-615f2fa07454-serviceca\") pod \"node-ca-db7tz\" (UID: \"5934c36a-2955-4f0c-a8e8-615f2fa07454\") " pod="openshift-image-registry/node-ca-db7tz" Oct 30 21:27:42.027148 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.026778 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2ae36e61-388d-4bc0-8810-aae71f3703c6-iptables-alerter-script\") pod \"iptables-alerter-wd8lr\" (UID: \"2ae36e61-388d-4bc0-8810-aae71f3703c6\") " pod="openshift-network-operator/iptables-alerter-wd8lr" Oct 30 21:27:42.034316 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.034298 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx5l7\" (UniqueName: \"kubernetes.io/projected/5934c36a-2955-4f0c-a8e8-615f2fa07454-kube-api-access-mx5l7\") pod \"node-ca-db7tz\" (UID: \"5934c36a-2955-4f0c-a8e8-615f2fa07454\") " pod="openshift-image-registry/node-ca-db7tz" Oct 30 21:27:42.044782 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.044762 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gstg\" (UniqueName: \"kubernetes.io/projected/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-kube-api-access-4gstg\") pod \"network-metrics-daemon-4r4zb\" (UID: \"3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78\") " pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:27:42.044850 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.044797 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwp8n\" (UniqueName: \"kubernetes.io/projected/2ae36e61-388d-4bc0-8810-aae71f3703c6-kube-api-access-vwp8n\") pod \"iptables-alerter-wd8lr\" (UID: \"2ae36e61-388d-4bc0-8810-aae71f3703c6\") " pod="openshift-network-operator/iptables-alerter-wd8lr" Oct 30 21:27:42.051142 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.051104 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjtk6\" (UniqueName: \"kubernetes.io/projected/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-kube-api-access-qjtk6\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:27:42.101034 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.101010 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:27:42.111258 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:42.111236 2543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846a43d3_aaa8_4d0f_bc4f_32916a05fb9a.slice/crio-445435dc345cd2db53684146141b296123e7565d34a25a5b484e87e56047dc39 WatchSource:0}: Error finding container 445435dc345cd2db53684146141b296123e7565d34a25a5b484e87e56047dc39: Status 404 returned error can't find the container with id 445435dc345cd2db53684146141b296123e7565d34a25a5b484e87e56047dc39 Oct 30 21:27:42.121955 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.121932 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:27:42.125909 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.125880 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j2j9f" Oct 30 21:27:42.132551 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:42.132528 2543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf15b3a1_2cbb_4eb2_8ac0_7bdf39fc72c4.slice/crio-1f3721f94fe154516ba62cca3e2cbe1788b0776dd7701f70a95539931b316fef WatchSource:0}: Error finding container 1f3721f94fe154516ba62cca3e2cbe1788b0776dd7701f70a95539931b316fef: Status 404 returned error can't find the container with id 1f3721f94fe154516ba62cca3e2cbe1788b0776dd7701f70a95539931b316fef Oct 30 21:27:42.134827 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:42.134805 2543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3af705e_b3ba_4379_8214_07ba1b431824.slice/crio-01498921daece598a7ce2734824ebc2fd4b2b973c4e826a4e09f9a58838d32b9 WatchSource:0}: Error finding container 01498921daece598a7ce2734824ebc2fd4b2b973c4e826a4e09f9a58838d32b9: Status 404 returned error can't find the container with id 01498921daece598a7ce2734824ebc2fd4b2b973c4e826a4e09f9a58838d32b9 Oct 30 21:27:42.135514 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.135495 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hl8qs" Oct 30 21:27:42.138762 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.138730 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:27:42.140976 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.140956 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7d6nt" Oct 30 21:27:42.145615 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:42.145596 2543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba16fc2_45ad_4ae0_b077_b4fc6bd7dda8.slice/crio-34905c4138e89ebabc77c38a62d4d277a3e9ad38e44c156e43ee57012e0deae7 WatchSource:0}: Error finding container 34905c4138e89ebabc77c38a62d4d277a3e9ad38e44c156e43ee57012e0deae7: Status 404 returned error can't find the container with id 34905c4138e89ebabc77c38a62d4d277a3e9ad38e44c156e43ee57012e0deae7 Oct 30 21:27:42.145878 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.145862 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wd8lr" Oct 30 21:27:42.148734 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:42.148708 2543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6841b7e6_42ec_493f_9613_ac4902212fac.slice/crio-69334d317041ccb071934aed82b2a2668b57f54889017d16c8443ff3bfdaca9e WatchSource:0}: Error finding container 69334d317041ccb071934aed82b2a2668b57f54889017d16c8443ff3bfdaca9e: Status 404 returned error can't find the container with id 69334d317041ccb071934aed82b2a2668b57f54889017d16c8443ff3bfdaca9e Oct 30 21:27:42.151519 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:42.151499 2543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950f0225_e1c9_4c9c_8fc6_25dba5db7120.slice/crio-2fe33a545cb5bbcb29c98119be301ae623972d70eff09fbb658bc8dd3ecd78c5 WatchSource:0}: Error finding container 2fe33a545cb5bbcb29c98119be301ae623972d70eff09fbb658bc8dd3ecd78c5: Status 404 returned error can't find the container with id 2fe33a545cb5bbcb29c98119be301ae623972d70eff09fbb658bc8dd3ecd78c5 Oct 30 21:27:42.155390 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:42.155373 2543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae36e61_388d_4bc0_8810_aae71f3703c6.slice/crio-46ea9f9182e771ae4b2a53ab78cc99399a8f3eba9f63301332e3064c96a24c69 WatchSource:0}: Error finding container 46ea9f9182e771ae4b2a53ab78cc99399a8f3eba9f63301332e3064c96a24c69: Status 404 returned error can't find the container with id 46ea9f9182e771ae4b2a53ab78cc99399a8f3eba9f63301332e3064c96a24c69 Oct 30 21:27:42.157708 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.157677 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:27:42.159995 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.159976 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-db7tz" Oct 30 21:27:42.166866 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:42.166846 2543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b83dac_47b9_4fb6_b39b_5e2dd2850c2d.slice/crio-acb30153b006b8ad51cb2725eeeb41d1406ddabeab22c41da892be6125a8d85a WatchSource:0}: Error finding container acb30153b006b8ad51cb2725eeeb41d1406ddabeab22c41da892be6125a8d85a: Status 404 returned error can't find the container with id acb30153b006b8ad51cb2725eeeb41d1406ddabeab22c41da892be6125a8d85a Oct 30 21:27:42.170648 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:27:42.170631 2543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5934c36a_2955_4f0c_a8e8_615f2fa07454.slice/crio-4219a488b10a8b86b8a9c029c3d199ff40e2ffda7684bf1ea381382f34a1023e WatchSource:0}: Error finding container 4219a488b10a8b86b8a9c029c3d199ff40e2ffda7684bf1ea381382f34a1023e: Status 404 returned error can't find the container with id 4219a488b10a8b86b8a9c029c3d199ff40e2ffda7684bf1ea381382f34a1023e Oct 30 21:27:42.429874 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.429843 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqcwl\" (UniqueName: \"kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl\") pod \"network-check-target-kv4cs\" (UID: \"bb88c1ff-4b28-4812-9b2b-8876983720e5\") " pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:27:42.430027 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:42.430006 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:27:42.430085 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:42.430031 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:27:42.430085 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:42.430045 2543 projected.go:196] Error preparing data for projected volume kube-api-access-qqcwl for pod openshift-network-diagnostics/network-check-target-kv4cs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:27:42.430149 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:42.430091 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl podName:bb88c1ff-4b28-4812-9b2b-8876983720e5 nodeName:}" failed. No retries permitted until 2026-10-30 21:27:43.430078142 +0000 UTC m=+4.019456291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qqcwl" (UniqueName: "kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl") pod "network-check-target-kv4cs" (UID: "bb88c1ff-4b28-4812-9b2b-8876983720e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:27:42.531000 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.530969 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs\") pod \"network-metrics-daemon-4r4zb\" (UID: \"3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78\") " pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:27:42.531157 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.531014 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret\") pod \"global-pull-secret-syncer-c2vrg\" (UID: \"183318eb-952e-4e93-b716-6ab50b9c58dd\") " pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:27:42.531157 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:42.531122 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:27:42.531157 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:42.531130 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:27:42.531313 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:42.531192 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret podName:183318eb-952e-4e93-b716-6ab50b9c58dd nodeName:}" failed. No retries permitted until 2026-10-30 21:27:43.531173479 +0000 UTC m=+4.120551643 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret") pod "global-pull-secret-syncer-c2vrg" (UID: "183318eb-952e-4e93-b716-6ab50b9c58dd") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:27:42.531313 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:42.531208 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs podName:3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78 nodeName:}" failed. No retries permitted until 2026-10-30 21:27:43.531199924 +0000 UTC m=+4.120578072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs") pod "network-metrics-daemon-4r4zb" (UID: "3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:27:42.948900 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.948763 2543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:22:41 +0000 UTC" deadline="2028-05-28 04:46:37.25844748 +0000 UTC" Oct 30 21:27:42.948900 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.948816 2543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13807h18m54.309635076s" Oct 30 21:27:42.982201 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.982174 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:27:42.982329 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:42.982297 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zb" podUID="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" Oct 30 21:27:42.982715 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:42.982693 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:27:42.982848 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:42.982811 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kv4cs" podUID="bb88c1ff-4b28-4812-9b2b-8876983720e5" Oct 30 21:27:43.011082 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:43.010988 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6fx47" event={"ID":"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a","Type":"ContainerStarted","Data":"445435dc345cd2db53684146141b296123e7565d34a25a5b484e87e56047dc39"} Oct 30 21:27:43.020662 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:43.020111 2543 generic.go:356] "Generic (PLEG): container finished" podID="40b5c76fae82f6d7a38a1a745d5d3947" containerID="15d831d327b6d46836a138b6aa732eecca3a25e6333f65eada26bf0d1f5095c0" exitCode=0 Oct 30 21:27:43.020662 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:43.020249 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" event={"ID":"40b5c76fae82f6d7a38a1a745d5d3947","Type":"ContainerDied","Data":"15d831d327b6d46836a138b6aa732eecca3a25e6333f65eada26bf0d1f5095c0"} Oct 30 21:27:43.022969 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:43.022904 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-db7tz" event={"ID":"5934c36a-2955-4f0c-a8e8-615f2fa07454","Type":"ContainerStarted","Data":"4219a488b10a8b86b8a9c029c3d199ff40e2ffda7684bf1ea381382f34a1023e"} Oct 30 21:27:43.025811 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:43.025716 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wd8lr" event={"ID":"2ae36e61-388d-4bc0-8810-aae71f3703c6","Type":"ContainerStarted","Data":"46ea9f9182e771ae4b2a53ab78cc99399a8f3eba9f63301332e3064c96a24c69"} Oct 30 21:27:43.039424 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:43.039349 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" event={"ID":"6841b7e6-42ec-493f-9613-ac4902212fac","Type":"ContainerStarted","Data":"69334d317041ccb071934aed82b2a2668b57f54889017d16c8443ff3bfdaca9e"} Oct 30 21:27:43.050742 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:43.050712 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" event={"ID":"3b404058373133d538844156a64ac398","Type":"ContainerStarted","Data":"4c876390172c7f6471923b80845c336ad6f3a26b2ef16272050dcf639ef92a58"} Oct 30 21:27:43.053660 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:43.053608 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" event={"ID":"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d","Type":"ContainerStarted","Data":"acb30153b006b8ad51cb2725eeeb41d1406ddabeab22c41da892be6125a8d85a"} Oct 30 21:27:43.059384 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:43.059359 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7d6nt" event={"ID":"950f0225-e1c9-4c9c-8fc6-25dba5db7120","Type":"ContainerStarted","Data":"2fe33a545cb5bbcb29c98119be301ae623972d70eff09fbb658bc8dd3ecd78c5"} Oct 30 21:27:43.068779 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:43.068359 2543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" podStartSLOduration=2.068343849 podStartE2EDuration="2.068343849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:43.067922753 +0000 UTC m=+3.657300922" watchObservedRunningTime="2026-10-30 21:27:43.068343849 +0000 UTC m=+3.657722023" Oct 30 21:27:43.090485 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:43.090426 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hl8qs" event={"ID":"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8","Type":"ContainerStarted","Data":"34905c4138e89ebabc77c38a62d4d277a3e9ad38e44c156e43ee57012e0deae7"} Oct 30 21:27:43.096465 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:43.096408 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j2j9f" event={"ID":"c3af705e-b3ba-4379-8214-07ba1b431824","Type":"ContainerStarted","Data":"01498921daece598a7ce2734824ebc2fd4b2b973c4e826a4e09f9a58838d32b9"} Oct 30 21:27:43.099340 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:43.099273 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcn8r" event={"ID":"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4","Type":"ContainerStarted","Data":"1f3721f94fe154516ba62cca3e2cbe1788b0776dd7701f70a95539931b316fef"} Oct 30 21:27:43.438405 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:43.438371 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqcwl\" (UniqueName: \"kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl\") pod \"network-check-target-kv4cs\" (UID: \"bb88c1ff-4b28-4812-9b2b-8876983720e5\") " pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:27:43.439305 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:43.438598 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:27:43.439305 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:43.438619 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:27:43.439305 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:43.438631 2543 projected.go:196] Error preparing data for projected volume kube-api-access-qqcwl for pod openshift-network-diagnostics/network-check-target-kv4cs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:27:43.439305 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:43.438685 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl podName:bb88c1ff-4b28-4812-9b2b-8876983720e5 nodeName:}" failed. No retries permitted until 2026-10-30 21:27:45.438667645 +0000 UTC m=+6.028045802 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qqcwl" (UniqueName: "kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl") pod "network-check-target-kv4cs" (UID: "bb88c1ff-4b28-4812-9b2b-8876983720e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:27:43.539511 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:43.539445 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret\") pod \"global-pull-secret-syncer-c2vrg\" (UID: \"183318eb-952e-4e93-b716-6ab50b9c58dd\") " pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:27:43.539646 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:43.539539 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs\") pod \"network-metrics-daemon-4r4zb\" (UID: \"3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78\") " pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:27:43.539703 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:43.539653 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:27:43.539781 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:43.539712 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs podName:3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78 nodeName:}" failed. No retries permitted until 2026-10-30 21:27:45.539693007 +0000 UTC m=+6.129071161 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs") pod "network-metrics-daemon-4r4zb" (UID: "3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:27:43.540099 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:43.540081 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:27:43.540180 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:43.540133 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret podName:183318eb-952e-4e93-b716-6ab50b9c58dd nodeName:}" failed. No retries permitted until 2026-10-30 21:27:45.540118223 +0000 UTC m=+6.129496379 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret") pod "global-pull-secret-syncer-c2vrg" (UID: "183318eb-952e-4e93-b716-6ab50b9c58dd") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:27:43.949351 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:43.949238 2543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:22:41 +0000 UTC" deadline="2028-04-28 07:10:03.844785206 +0000 UTC" Oct 30 21:27:43.949351 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:43.949307 2543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13089h42m19.89548196s" Oct 30 21:27:43.983560 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:43.983531 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:27:43.983724 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:43.983674 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c2vrg" podUID="183318eb-952e-4e93-b716-6ab50b9c58dd" Oct 30 21:27:44.138464 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:44.137847 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" event={"ID":"40b5c76fae82f6d7a38a1a745d5d3947","Type":"ContainerStarted","Data":"a6acde858f50a052f287c1205b925236d7ad4dd9752fb2b0fbd2b0424676fad5"} Oct 30 21:27:44.982361 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:44.982318 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:27:44.982913 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:44.982446 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kv4cs" podUID="bb88c1ff-4b28-4812-9b2b-8876983720e5" Oct 30 21:27:44.982913 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:44.982874 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:27:44.983040 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:44.982962 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zb" podUID="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" Oct 30 21:27:45.154314 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:45.154277 2543 generic.go:356] "Generic (PLEG): container finished" podID="cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4" containerID="2e524e22ab8a5f1c5fbd8662c4aa5447e439efd936a6720704a4163d72f7edb9" exitCode=0 Oct 30 21:27:45.155355 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:45.155323 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcn8r" event={"ID":"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4","Type":"ContainerDied","Data":"2e524e22ab8a5f1c5fbd8662c4aa5447e439efd936a6720704a4163d72f7edb9"} Oct 30 21:27:45.184676 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:45.184631 2543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" podStartSLOduration=4.18461408 podStartE2EDuration="4.18461408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:44.156931548 +0000 UTC m=+4.746309723" watchObservedRunningTime="2026-10-30 21:27:45.18461408 +0000 UTC m=+5.773992253" Oct 30 21:27:45.455322 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:45.455244 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqcwl\" (UniqueName: \"kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl\") pod \"network-check-target-kv4cs\" (UID: \"bb88c1ff-4b28-4812-9b2b-8876983720e5\") " pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:27:45.455475 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:45.455410 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:27:45.455475 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:45.455430 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:27:45.455475 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:45.455443 2543 projected.go:196] Error preparing data for projected volume kube-api-access-qqcwl for pod openshift-network-diagnostics/network-check-target-kv4cs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:27:45.455634 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:45.455502 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl podName:bb88c1ff-4b28-4812-9b2b-8876983720e5 nodeName:}" failed. No retries permitted until 2026-10-30 21:27:49.455482719 +0000 UTC m=+10.044860875 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qqcwl" (UniqueName: "kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl") pod "network-check-target-kv4cs" (UID: "bb88c1ff-4b28-4812-9b2b-8876983720e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:27:45.556360 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:45.556320 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret\") pod \"global-pull-secret-syncer-c2vrg\" (UID: \"183318eb-952e-4e93-b716-6ab50b9c58dd\") " pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:27:45.556575 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:45.556420 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs\") pod \"network-metrics-daemon-4r4zb\" (UID: \"3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78\") " pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:27:45.556575 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:45.556517 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:27:45.556811 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:45.556578 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs podName:3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78 nodeName:}" failed. No retries permitted until 2026-10-30 21:27:49.55655918 +0000 UTC m=+10.145937353 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs") pod "network-metrics-daemon-4r4zb" (UID: "3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:27:45.556811 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:45.556637 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:27:45.556811 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:45.556662 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret podName:183318eb-952e-4e93-b716-6ab50b9c58dd nodeName:}" failed. No retries permitted until 2026-10-30 21:27:49.55665497 +0000 UTC m=+10.146033121 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret") pod "global-pull-secret-syncer-c2vrg" (UID: "183318eb-952e-4e93-b716-6ab50b9c58dd") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:27:45.983298 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:45.983265 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:27:45.983748 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:45.983408 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c2vrg" podUID="183318eb-952e-4e93-b716-6ab50b9c58dd" Oct 30 21:27:46.175790 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:46.175690 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7d6nt" event={"ID":"950f0225-e1c9-4c9c-8fc6-25dba5db7120","Type":"ContainerStarted","Data":"96cc0c90be54bf2d62bf836f7e459e58c788dbfc8a8a5b966f1cf0b7fac9275d"} Oct 30 21:27:46.982284 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:46.982215 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:27:46.982449 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:46.982353 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zb" podUID="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" Oct 30 21:27:46.982780 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:46.982711 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:27:46.982894 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:46.982822 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kv4cs" podUID="bb88c1ff-4b28-4812-9b2b-8876983720e5" Oct 30 21:27:47.182606 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:47.182569 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-db7tz" event={"ID":"5934c36a-2955-4f0c-a8e8-615f2fa07454","Type":"ContainerStarted","Data":"55c6cb60e4c4ecb934f19932101f43ee78b3b8d6d4685db2224b9c8cf0180c56"} Oct 30 21:27:47.202189 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:47.202132 2543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7d6nt" podStartSLOduration=3.4984975560000002 podStartE2EDuration="7.202114067s" totalImagesPullingTime="3.703616511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:46.194510698 +0000 UTC m=+6.783888867" watchObservedRunningTime="2026-10-30 21:27:47.202114067 +0000 UTC m=+7.791492239" Oct 30 21:27:47.984637 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:47.984160 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:27:47.984637 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:47.984280 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c2vrg" podUID="183318eb-952e-4e93-b716-6ab50b9c58dd" Oct 30 21:27:48.188300 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:48.188232 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" event={"ID":"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d","Type":"ContainerStarted","Data":"9aa004b08c16892708f5300d1d4996a24d87fd0f5efe0e9dba069869da773c56"} Oct 30 21:27:48.956352 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:48.956281 2543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7d6nt" Oct 30 21:27:48.957982 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:48.957779 2543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7d6nt" Oct 30 21:27:48.977994 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:48.977122 2543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-db7tz" podStartSLOduration=4.416811632 podStartE2EDuration="8.977105576s" totalImagesPullingTime="4.560293944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:27:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:47.208967624 +0000 UTC m=+7.798345799" watchObservedRunningTime="2026-10-30 21:27:48.977105576 +0000 UTC m=+9.566483749" Oct 30 21:27:48.982026 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:48.981413 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:27:48.982026 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:48.981528 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zb" podUID="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" Oct 30 21:27:48.982026 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:48.981851 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:27:48.982026 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:48.981954 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kv4cs" podUID="bb88c1ff-4b28-4812-9b2b-8876983720e5" Oct 30 21:27:49.191433 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:49.191408 2543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7d6nt" Oct 30 21:27:49.191967 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:49.191949 2543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7d6nt" Oct 30 21:27:49.320798 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:49.320445 2543 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:27:49.485496 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:49.485411 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqcwl\" (UniqueName: \"kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl\") pod \"network-check-target-kv4cs\" (UID: \"bb88c1ff-4b28-4812-9b2b-8876983720e5\") " pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:27:49.485718 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:49.485699 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:27:49.485831 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:49.485727 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:27:49.485831 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:49.485740 2543 projected.go:196] Error preparing data for projected volume kube-api-access-qqcwl for pod openshift-network-diagnostics/network-check-target-kv4cs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:27:49.486846 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:49.486815 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl podName:bb88c1ff-4b28-4812-9b2b-8876983720e5 nodeName:}" failed. No retries permitted until 2026-10-30 21:27:57.485795382 +0000 UTC m=+18.075173534 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qqcwl" (UniqueName: "kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl") pod "network-check-target-kv4cs" (UID: "bb88c1ff-4b28-4812-9b2b-8876983720e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:27:49.586997 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:49.586255 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs\") pod \"network-metrics-daemon-4r4zb\" (UID: \"3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78\") " pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:27:49.586997 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:49.586306 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret\") pod \"global-pull-secret-syncer-c2vrg\" (UID: \"183318eb-952e-4e93-b716-6ab50b9c58dd\") " pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:27:49.586997 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:49.586424 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:27:49.586997 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:49.586468 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret podName:183318eb-952e-4e93-b716-6ab50b9c58dd nodeName:}" failed. No retries permitted until 2026-10-30 21:27:57.586456982 +0000 UTC m=+18.175835138 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret") pod "global-pull-secret-syncer-c2vrg" (UID: "183318eb-952e-4e93-b716-6ab50b9c58dd") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:27:49.586997 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:49.586872 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:27:49.586997 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:49.586925 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs podName:3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78 nodeName:}" failed. No retries permitted until 2026-10-30 21:27:57.586909605 +0000 UTC m=+18.176287757 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs") pod "network-metrics-daemon-4r4zb" (UID: "3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:27:49.982807 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:49.982780 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:27:49.982941 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:49.982918 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c2vrg" podUID="183318eb-952e-4e93-b716-6ab50b9c58dd" Oct 30 21:27:50.017782 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:50.015828 2543 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:27:49.320465059Z","UUID":"a75b0eb5-b7c8-4310-8b85-fd75a44e330f","Handler":null,"Name":"","Endpoint":""} Oct 30 21:27:50.022013 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:50.019733 2543 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:27:50.023861 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:50.023843 2543 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:27:50.194514 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:50.194483 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hl8qs" event={"ID":"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8","Type":"ContainerStarted","Data":"1040b7535e035f0ca47daac0a76458337f4f47eb3b85ee36c3c8b34524842b5e"} Oct 30 21:27:50.197698 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:50.197473 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j2j9f" event={"ID":"c3af705e-b3ba-4379-8214-07ba1b431824","Type":"ContainerStarted","Data":"06734a18a7b492fec36c5455574b548654ae8d98338bb6601a38107c5469c366"} Oct 30 21:27:50.199932 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:50.199907 2543 generic.go:356] "Generic (PLEG): container finished" podID="cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4" containerID="b30bd30e30ec76492ef051e7c965b9db53d6897a56304b806a5e42db46c3233c" exitCode=0 Oct 30 21:27:50.200031 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:50.199939 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcn8r" event={"ID":"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4","Type":"ContainerDied","Data":"b30bd30e30ec76492ef051e7c965b9db53d6897a56304b806a5e42db46c3233c"} Oct 30 21:27:50.204185 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:50.204163 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" event={"ID":"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d","Type":"ContainerStarted","Data":"3dd12149502535c8ecf8b417d2e106b9372f61127404ba02709cd3fd998a3cc7"} Oct 30 21:27:50.217059 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:50.217014 2543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-hl8qs" podStartSLOduration=3.7313486559999998 podStartE2EDuration="11.216999424s" totalImagesPullingTime="7.485650768s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:50.216989721 +0000 UTC m=+10.806367898" watchObservedRunningTime="2026-10-30 21:27:50.216999424 +0000 UTC m=+10.806377598" Oct 30 21:27:50.233716 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:50.233675 2543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-j2j9f" podStartSLOduration=3.741337596 podStartE2EDuration="11.233667254s" totalImagesPullingTime="7.492329658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:50.233572543 +0000 UTC m=+10.822950719" watchObservedRunningTime="2026-10-30 21:27:50.233667254 +0000 UTC m=+10.823045425" Oct 30 21:27:50.982026 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:50.982001 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:27:50.982109 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:50.982025 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:27:50.982158 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:50.982121 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kv4cs" podUID="bb88c1ff-4b28-4812-9b2b-8876983720e5" Oct 30 21:27:50.982282 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:50.982256 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zb" podUID="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" Oct 30 21:27:51.209406 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:51.209371 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" event={"ID":"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d","Type":"ContainerStarted","Data":"300a8a4364f3df1643db62922d3219d66e7281c8c0fb459c7bc1a9c145c320f0"} Oct 30 21:27:51.212614 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:51.212587 2543 generic.go:356] "Generic (PLEG): container finished" podID="cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4" containerID="6d58dd436714e080ad79a776dccae59c9cb254adae1ff1f60ce897d56c9154e6" exitCode=0 Oct 30 21:27:51.212770 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:51.212681 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcn8r" event={"ID":"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4","Type":"ContainerDied","Data":"6d58dd436714e080ad79a776dccae59c9cb254adae1ff1f60ce897d56c9154e6"} Oct 30 21:27:51.238093 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:51.238044 2543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" podStartSLOduration=3.396382932 podStartE2EDuration="11.238027382s" totalImagesPullingTime="7.84164445s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:27:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:51.237852739 +0000 UTC m=+11.827230912" watchObservedRunningTime="2026-10-30 21:27:51.238027382 +0000 UTC m=+11.827405556" Oct 30 21:27:51.981602 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:51.981570 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:27:51.981793 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:51.981703 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c2vrg" podUID="183318eb-952e-4e93-b716-6ab50b9c58dd" Oct 30 21:27:52.216638 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:52.216579 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6fx47" event={"ID":"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a","Type":"ContainerStarted","Data":"5e8ab522710e0a01471455e1dde43385cf549d2b3908994669a81e5af8907f76"} Oct 30 21:27:52.218401 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:52.218377 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wd8lr" event={"ID":"2ae36e61-388d-4bc0-8810-aae71f3703c6","Type":"ContainerStarted","Data":"ec024cde6ab43521b117c236d1636a49a33c04d881857a084ccf14f781316935"} Oct 30 21:27:52.236206 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:52.236130 2543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wd8lr" podStartSLOduration=4.59161319 podStartE2EDuration="12.236115834s" totalImagesPullingTime="7.644502644s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:27:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:52.236075104 +0000 UTC m=+12.825453276" watchObservedRunningTime="2026-10-30 21:27:52.236115834 +0000 UTC m=+12.825494007" Oct 30 21:27:52.981924 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:52.981853 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:27:52.982065 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:52.981969 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kv4cs" podUID="bb88c1ff-4b28-4812-9b2b-8876983720e5" Oct 30 21:27:52.982065 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:52.982030 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:27:52.982154 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:52.982132 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zb" podUID="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" Oct 30 21:27:53.222913 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:53.222876 2543 generic.go:356] "Generic (PLEG): container finished" podID="cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4" containerID="41f6bac12e157d04e12e48956ae478191f187cd5d3125e78d1debd21784dae77" exitCode=0 Oct 30 21:27:53.223372 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:53.222957 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcn8r" event={"ID":"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4","Type":"ContainerDied","Data":"41f6bac12e157d04e12e48956ae478191f187cd5d3125e78d1debd21784dae77"} Oct 30 21:27:53.269794 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:53.269698 2543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6fx47" podStartSLOduration=4.258400665 podStartE2EDuration="14.269685848s" totalImagesPullingTime="10.011285183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:53.269448366 +0000 UTC m=+13.858826539" watchObservedRunningTime="2026-10-30 21:27:53.269685848 +0000 UTC m=+13.859064019" Oct 30 21:27:53.982251 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:53.982049 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:27:53.982509 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:53.982479 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c2vrg" podUID="183318eb-952e-4e93-b716-6ab50b9c58dd" Oct 30 21:27:54.982178 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:54.982148 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:27:54.982178 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:54.982185 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:27:54.982672 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:54.982277 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kv4cs" podUID="bb88c1ff-4b28-4812-9b2b-8876983720e5" Oct 30 21:27:54.982672 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:54.982432 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zb" podUID="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" Oct 30 21:27:55.722998 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:55.722904 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:55.911235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:55.911203 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:55.984275 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:55.984222 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:27:55.984594 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:55.984317 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c2vrg" podUID="183318eb-952e-4e93-b716-6ab50b9c58dd" Oct 30 21:27:56.231949 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:56.231919 2543 generic.go:356] "Generic (PLEG): container finished" podID="cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4" containerID="4c55c9fac74c10a75c547a362a820f150b8d5a5796bccaeb9a3252d592c0d800" exitCode=0 Oct 30 21:27:56.232080 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:56.231965 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcn8r" event={"ID":"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4","Type":"ContainerDied","Data":"4c55c9fac74c10a75c547a362a820f150b8d5a5796bccaeb9a3252d592c0d800"} Oct 30 21:27:56.981748 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:56.981719 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:27:56.981945 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:56.981911 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zb" podUID="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" Oct 30 21:27:56.982000 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:56.981953 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:27:56.982045 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:56.982028 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kv4cs" podUID="bb88c1ff-4b28-4812-9b2b-8876983720e5" Oct 30 21:27:57.236544 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:57.236472 2543 generic.go:356] "Generic (PLEG): container finished" podID="cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4" containerID="794f49925c3ae47119ef384cb7faea61a69f9a39e498f2deb87727553d4b6477" exitCode=0 Oct 30 21:27:57.236984 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:57.236557 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcn8r" event={"ID":"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4","Type":"ContainerDied","Data":"794f49925c3ae47119ef384cb7faea61a69f9a39e498f2deb87727553d4b6477"} Oct 30 21:27:57.239463 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:57.239446 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" event={"ID":"6841b7e6-42ec-493f-9613-ac4902212fac","Type":"ContainerStarted","Data":"906b795e9e71755daf7cd2e4b49a2cbebf62d23dc591a68e1efd033806de9236"} Oct 30 21:27:57.239524 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:57.239506 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" event={"ID":"6841b7e6-42ec-493f-9613-ac4902212fac","Type":"ContainerStarted","Data":"c29cdc6ff8f68929a3f6193152f45225a2f08b6847e5b36d4022f0f22058659c"} Oct 30 21:27:57.239559 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:57.239538 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" event={"ID":"6841b7e6-42ec-493f-9613-ac4902212fac","Type":"ContainerStarted","Data":"82e0341d681620577a4b4cec8b89c6fd6aec0e9fe49da495a35eb1e4f9d0063d"} Oct 30 21:27:57.239588 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:57.239572 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" event={"ID":"6841b7e6-42ec-493f-9613-ac4902212fac","Type":"ContainerStarted","Data":"bb9c9c59fadd775bea184275e9d616fbecce43610e0594571b6ba2d9ddddd1d7"} Oct 30 21:27:57.239618 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:57.239599 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" event={"ID":"6841b7e6-42ec-493f-9613-ac4902212fac","Type":"ContainerStarted","Data":"3ab5d212d3f963e8dffad8a7205a119be94867a4c75dec13c6c742b89172c2c6"} Oct 30 21:27:57.239650 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:57.239632 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" event={"ID":"6841b7e6-42ec-493f-9613-ac4902212fac","Type":"ContainerStarted","Data":"bd8d456dcfbc8815026ddfe0e7373d7bbaed82d6be0b0d7e8a71388ebee3d9ec"} Oct 30 21:27:57.262904 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:57.262884 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:57.546722 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:57.546648 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqcwl\" (UniqueName: \"kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl\") pod \"network-check-target-kv4cs\" (UID: \"bb88c1ff-4b28-4812-9b2b-8876983720e5\") " pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:27:57.546843 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:57.546797 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:27:57.546843 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:57.546814 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:27:57.546843 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:57.546824 2543 projected.go:196] Error preparing data for projected volume kube-api-access-qqcwl for pod openshift-network-diagnostics/network-check-target-kv4cs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:27:57.546950 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:57.546867 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl podName:bb88c1ff-4b28-4812-9b2b-8876983720e5 nodeName:}" failed. No retries permitted until 2026-10-30 21:28:13.546854226 +0000 UTC m=+34.136232375 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qqcwl" (UniqueName: "kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl") pod "network-check-target-kv4cs" (UID: "bb88c1ff-4b28-4812-9b2b-8876983720e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:27:57.647638 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:57.647613 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs\") pod \"network-metrics-daemon-4r4zb\" (UID: \"3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78\") " pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:27:57.647742 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:57.647644 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret\") pod \"global-pull-secret-syncer-c2vrg\" (UID: \"183318eb-952e-4e93-b716-6ab50b9c58dd\") " pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:27:57.647742 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:57.647736 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:27:57.647867 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:57.647771 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:27:57.647867 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:57.647792 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret podName:183318eb-952e-4e93-b716-6ab50b9c58dd nodeName:}" failed. No retries permitted until 2026-10-30 21:28:13.647781012 +0000 UTC m=+34.237159162 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret") pod "global-pull-secret-syncer-c2vrg" (UID: "183318eb-952e-4e93-b716-6ab50b9c58dd") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:27:57.647867 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:57.647825 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs podName:3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78 nodeName:}" failed. No retries permitted until 2026-10-30 21:28:13.64780813 +0000 UTC m=+34.237186294 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs") pod "network-metrics-daemon-4r4zb" (UID: "3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:27:57.982152 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:57.982091 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:27:57.982267 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:57.982183 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c2vrg" podUID="183318eb-952e-4e93-b716-6ab50b9c58dd" Oct 30 21:27:58.244727 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:58.244650 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcn8r" event={"ID":"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4","Type":"ContainerStarted","Data":"ddd9dfe6805c271027ce9853fa308ad1c3a4b8d92991226bc83e93856dec00f0"} Oct 30 21:27:58.272396 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:58.272350 2543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vcn8r" podStartSLOduration=8.092603405 podStartE2EDuration="19.272338691s" totalImagesPullingTime="11.179735286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:27:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:58.271931241 +0000 UTC m=+18.861309413" watchObservedRunningTime="2026-10-30 21:27:58.272338691 +0000 UTC m=+18.861716863" Oct 30 21:27:58.981658 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:58.981632 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:27:58.981847 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:58.981776 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:27:58.981847 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:58.981777 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kv4cs" podUID="bb88c1ff-4b28-4812-9b2b-8876983720e5" Oct 30 21:27:58.981966 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:58.981879 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zb" podUID="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" Oct 30 21:27:59.138681 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:59.138604 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:59.231769 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:59.231721 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:59.250357 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:59.250324 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" event={"ID":"6841b7e6-42ec-493f-9613-ac4902212fac","Type":"ContainerStarted","Data":"a11f553958fc31dac37a0e17d29bb4d0fc75cc53740a65ee5f8837da95c2d434"} Oct 30 21:27:59.809733 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:59.809704 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:59.911804 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:59.911774 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:59.984838 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:27:59.984819 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:27:59.984923 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:27:59.984905 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c2vrg" podUID="183318eb-952e-4e93-b716-6ab50b9c58dd" Oct 30 21:28:00.981346 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:00.981327 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:28:00.981710 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:00.981568 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kv4cs" podUID="bb88c1ff-4b28-4812-9b2b-8876983720e5" Oct 30 21:28:00.981710 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:00.981597 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:28:00.981710 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:00.981679 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zb" podUID="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" Oct 30 21:28:01.257147 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:01.257039 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" event={"ID":"6841b7e6-42ec-493f-9613-ac4902212fac","Type":"ContainerStarted","Data":"c19ae78ec19b65e0664b70b027453a53592b1b81ac3dde562bfbb3c76538d432"} Oct 30 21:28:01.257337 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:01.257298 2543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:01.257461 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:01.257385 2543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:01.257461 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:01.257426 2543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:01.272711 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:01.272691 2543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:01.273529 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:01.273511 2543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:01.296332 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:01.294962 2543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" podStartSLOduration=7.92014595 podStartE2EDuration="22.294944493s" totalImagesPullingTime="14.374798543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:01.293998379 +0000 UTC m=+21.883376551" watchObservedRunningTime="2026-10-30 21:28:01.294944493 +0000 UTC m=+21.884322669" Oct 30 21:28:01.835152 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:01.835117 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:01.964235 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:01.964181 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:01.982303 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:01.982278 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:01.982638 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:01.982398 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c2vrg" podUID="183318eb-952e-4e93-b716-6ab50b9c58dd" Oct 30 21:28:02.079294 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:02.079269 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:02.381279 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:02.381251 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:02.505471 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:02.505434 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:02.651367 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:02.651312 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:02.773865 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:02.773828 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:02.860808 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:02.860770 2543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c2vrg"] Oct 30 21:28:02.860943 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:02.860872 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:02.860995 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:02.860960 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c2vrg" podUID="183318eb-952e-4e93-b716-6ab50b9c58dd" Oct 30 21:28:02.868518 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:02.868491 2543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kv4cs"] Oct 30 21:28:02.868644 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:02.868612 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:28:02.868738 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:02.868714 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kv4cs" podUID="bb88c1ff-4b28-4812-9b2b-8876983720e5" Oct 30 21:28:02.869244 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:02.869225 2543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4r4zb"] Oct 30 21:28:02.869332 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:02.869317 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:28:02.869424 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:02.869405 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zb" podUID="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" Oct 30 21:28:03.414359 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:03.414328 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:03.507545 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:03.507509 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:03.982134 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:03.982107 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:03.982349 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:03.982207 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c2vrg" podUID="183318eb-952e-4e93-b716-6ab50b9c58dd" Oct 30 21:28:04.650114 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:04.648010 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:04.823609 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:04.823577 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:04.940796 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:04.940700 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:04.981874 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:04.981838 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:28:04.982028 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:04.981934 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kv4cs" podUID="bb88c1ff-4b28-4812-9b2b-8876983720e5" Oct 30 21:28:04.982028 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:04.981988 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:28:04.982148 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:04.982079 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zb" podUID="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" Oct 30 21:28:05.050820 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:05.050786 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:05.180664 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:05.180632 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:05.310833 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:05.310809 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:05.510787 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:05.510748 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:05.550412 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:05.550388 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j2j9f_c3af705e-b3ba-4379-8214-07ba1b431824/dns-node-resolver/0.log" Oct 30 21:28:05.982411 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:05.982381 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:05.982809 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:05.982514 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c2vrg" podUID="183318eb-952e-4e93-b716-6ab50b9c58dd" Oct 30 21:28:06.447313 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:06.447280 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:06.539030 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:06.539005 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-db7tz_5934c36a-2955-4f0c-a8e8-615f2fa07454/node-ca/0.log" Oct 30 21:28:06.649459 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:06.649425 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:06.849425 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:06.849389 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:06.950357 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:06.950325 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:06.981445 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:06.981417 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:28:06.981572 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:06.981523 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kv4cs" podUID="bb88c1ff-4b28-4812-9b2b-8876983720e5" Oct 30 21:28:06.981619 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:06.981574 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:28:06.981686 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:06.981670 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zb" podUID="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" Oct 30 21:28:07.050228 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:07.050197 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:07.247012 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:07.246930 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:07.696504 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:07.696471 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:07.982151 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:07.982074 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:07.982297 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:07.982206 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c2vrg" podUID="183318eb-952e-4e93-b716-6ab50b9c58dd" Oct 30 21:28:08.036398 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:08.036370 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:08.477593 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:08.477557 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:08.658918 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:08.658887 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:08.893772 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:08.893734 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:08.982109 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:08.982075 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:28:08.982266 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:08.982126 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:28:08.982266 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:08.982214 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zb" podUID="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" Oct 30 21:28:08.982354 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:08.982311 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kv4cs" podUID="bb88c1ff-4b28-4812-9b2b-8876983720e5" Oct 30 21:28:09.100704 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:09.100669 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:09.266281 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:09.266200 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:09.396874 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:09.396835 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:09.500770 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:09.500733 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:09.643976 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:09.643945 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:09.755646 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:09.755618 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:09.856894 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:09.856858 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:09.983127 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:09.983055 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:09.983276 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:09.983136 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c2vrg" podUID="183318eb-952e-4e93-b716-6ab50b9c58dd" Oct 30 21:28:10.247232 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:10.247165 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:10.457444 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:10.457416 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:10.668009 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:10.667979 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:10.783704 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:10.783663 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:10.964740 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:10.964663 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:10.981840 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:10.981818 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:28:10.981941 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:10.981911 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kv4cs" podUID="bb88c1ff-4b28-4812-9b2b-8876983720e5" Oct 30 21:28:10.982000 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:10.981975 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:28:10.982098 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:10.982079 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zb" podUID="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" Oct 30 21:28:11.076937 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:11.076905 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:11.267089 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:11.267007 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:11.981439 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:11.981410 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:11.981934 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:11.981528 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c2vrg" podUID="183318eb-952e-4e93-b716-6ab50b9c58dd" Oct 30 21:28:12.117548 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:12.117517 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:12.456036 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:12.456005 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:12.555939 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:12.555908 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:12.659641 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:12.659612 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:12.760208 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:12.760134 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:12.860028 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:12.859993 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:12.981517 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:12.981483 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:28:12.981517 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:12.981514 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:28:12.982003 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:12.981597 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kv4cs" podUID="bb88c1ff-4b28-4812-9b2b-8876983720e5" Oct 30 21:28:12.982003 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:12.981673 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zb" podUID="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" Oct 30 21:28:13.050055 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:13.050030 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:13.248896 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:13.248867 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:13.570731 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:13.570700 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqcwl\" (UniqueName: \"kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl\") pod \"network-check-target-kv4cs\" (UID: \"bb88c1ff-4b28-4812-9b2b-8876983720e5\") " pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:28:13.570899 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:13.570839 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:28:13.570899 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:13.570854 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:28:13.570899 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:13.570863 2543 projected.go:196] Error preparing data for projected volume kube-api-access-qqcwl for pod openshift-network-diagnostics/network-check-target-kv4cs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:28:13.570995 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:13.570908 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl podName:bb88c1ff-4b28-4812-9b2b-8876983720e5 nodeName:}" failed. No retries permitted until 2026-10-30 21:28:45.570896436 +0000 UTC m=+66.160274587 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qqcwl" (UniqueName: "kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl") pod "network-check-target-kv4cs" (UID: "bb88c1ff-4b28-4812-9b2b-8876983720e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:28:13.659576 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:13.659545 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:13.671604 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:13.671578 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs\") pod \"network-metrics-daemon-4r4zb\" (UID: \"3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78\") " pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:28:13.671664 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:13.671615 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret\") pod \"global-pull-secret-syncer-c2vrg\" (UID: \"183318eb-952e-4e93-b716-6ab50b9c58dd\") " pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:13.671718 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:13.671706 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:28:13.671780 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:13.671770 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret podName:183318eb-952e-4e93-b716-6ab50b9c58dd nodeName:}" failed. No retries permitted until 2026-10-30 21:28:45.67174118 +0000 UTC m=+66.261119343 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret") pod "global-pull-secret-syncer-c2vrg" (UID: "183318eb-952e-4e93-b716-6ab50b9c58dd") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:28:13.671829 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:13.671709 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:28:13.671861 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:13.671840 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs podName:3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78 nodeName:}" failed. No retries permitted until 2026-10-30 21:28:45.671826498 +0000 UTC m=+66.261204653 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs") pod "network-metrics-daemon-4r4zb" (UID: "3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:28:13.759702 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:13.759672 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:13.982126 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:13.982052 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:13.982492 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:13.982162 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c2vrg" podUID="183318eb-952e-4e93-b716-6ab50b9c58dd" Oct 30 21:28:14.444956 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:14.444924 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:14.648600 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:14.648573 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:14.797377 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:14.797294 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:14.898355 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:14.898317 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:14.982335 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:14.982293 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:28:14.982335 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:14.982331 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:28:14.982901 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:14.982434 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zb" podUID="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" Oct 30 21:28:14.982901 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:14.982542 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kv4cs" podUID="bb88c1ff-4b28-4812-9b2b-8876983720e5" Oct 30 21:28:15.052810 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:15.052719 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:15.262208 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:15.262178 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:15.894353 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:15.894326 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:15.981640 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:15.981596 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:15.981872 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:15.981705 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c2vrg" podUID="183318eb-952e-4e93-b716-6ab50b9c58dd" Oct 30 21:28:16.061946 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:16.061917 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:16.981923 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:16.981885 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:28:16.981923 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:16.981922 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:28:16.982128 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:16.982007 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zb" podUID="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" Oct 30 21:28:16.982128 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:16.982109 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kv4cs" podUID="bb88c1ff-4b28-4812-9b2b-8876983720e5" Oct 30 21:28:17.981620 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:17.981586 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:17.982030 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:17.981716 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c2vrg" podUID="183318eb-952e-4e93-b716-6ab50b9c58dd" Oct 30 21:28:18.981643 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:18.981606 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:28:18.982117 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:18.981714 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zb" podUID="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" Oct 30 21:28:18.982117 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:18.981742 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:28:18.982117 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:18.981814 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kv4cs" podUID="bb88c1ff-4b28-4812-9b2b-8876983720e5" Oct 30 21:28:19.982900 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:19.982870 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:19.983357 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:19.982974 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c2vrg" podUID="183318eb-952e-4e93-b716-6ab50b9c58dd" Oct 30 21:28:20.981692 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:20.981648 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:28:20.981914 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:20.981698 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:28:20.981914 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:20.981812 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zb" podUID="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" Oct 30 21:28:20.982042 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:20.981920 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kv4cs" podUID="bb88c1ff-4b28-4812-9b2b-8876983720e5" Oct 30 21:28:21.982065 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:21.982026 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:21.982578 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:21.982135 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c2vrg" podUID="183318eb-952e-4e93-b716-6ab50b9c58dd" Oct 30 21:28:22.981723 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:22.981680 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:28:22.981879 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:22.981810 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kv4cs" podUID="bb88c1ff-4b28-4812-9b2b-8876983720e5" Oct 30 21:28:22.981879 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:22.981861 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:28:22.981965 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:22.981942 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zb" podUID="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" Oct 30 21:28:23.473018 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:23.472988 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:23.595286 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:23.595253 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:23.981673 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:23.981644 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:23.981863 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:23.981739 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c2vrg" podUID="183318eb-952e-4e93-b716-6ab50b9c58dd" Oct 30 21:28:24.981903 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:24.981858 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:28:24.982311 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:24.981858 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:28:24.982311 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:24.981981 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4r4zb" podUID="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" Oct 30 21:28:24.982311 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:24.982037 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kv4cs" podUID="bb88c1ff-4b28-4812-9b2b-8876983720e5" Oct 30 21:28:25.498880 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:25.498841 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:25.820500 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:25.820470 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:25.920415 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:25.920378 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:25.982367 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:25.982341 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:25.982877 ip-10-0-140-163 kubenswrapper[2543]: E1030 21:28:25.982495 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c2vrg" podUID="183318eb-952e-4e93-b716-6ab50b9c58dd" Oct 30 21:28:26.171423 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.171399 2543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeReady" Oct 30 21:28:26.171555 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.171508 2543 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:28:26.229742 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.229708 2543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wdr8m"] Oct 30 21:28:26.232595 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.232580 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wdr8m" Oct 30 21:28:26.237068 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.237041 2543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-99chm"] Oct 30 21:28:26.240107 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.240086 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-99chm" Oct 30 21:28:26.240813 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.240788 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:28:26.240894 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.240840 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-r96r5\"" Oct 30 21:28:26.243074 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.243059 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:28:26.243545 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.243532 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:28:26.252383 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.252362 2543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wdr8m"] Oct 30 21:28:26.253464 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.253442 2543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-99chm"] Oct 30 21:28:26.257478 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.257458 2543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-htw5p"] Oct 30 21:28:26.259164 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.259141 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c702aef-eccc-4da1-8040-c860c607a466-tmp-dir\") pod \"dns-default-99chm\" (UID: \"5c702aef-eccc-4da1-8040-c860c607a466\") " pod="openshift-dns/dns-default-99chm" Oct 30 21:28:26.259264 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.259182 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c702aef-eccc-4da1-8040-c860c607a466-metrics-tls\") pod \"dns-default-99chm\" (UID: \"5c702aef-eccc-4da1-8040-c860c607a466\") " pod="openshift-dns/dns-default-99chm" Oct 30 21:28:26.259328 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.259282 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ztrc\" (UniqueName: \"kubernetes.io/projected/808521e6-a0b3-4a24-b0b0-5a3cabffe0f5-kube-api-access-7ztrc\") pod \"ingress-canary-wdr8m\" (UID: \"808521e6-a0b3-4a24-b0b0-5a3cabffe0f5\") " pod="openshift-ingress-canary/ingress-canary-wdr8m" Oct 30 21:28:26.259383 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.259348 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/808521e6-a0b3-4a24-b0b0-5a3cabffe0f5-cert\") pod \"ingress-canary-wdr8m\" (UID: \"808521e6-a0b3-4a24-b0b0-5a3cabffe0f5\") " pod="openshift-ingress-canary/ingress-canary-wdr8m" Oct 30 21:28:26.259432 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.259387 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5c702aef-eccc-4da1-8040-c860c607a466-config-volume\") pod \"dns-default-99chm\" (UID: \"5c702aef-eccc-4da1-8040-c860c607a466\") " pod="openshift-dns/dns-default-99chm" Oct 30 21:28:26.259432 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.259421 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2vfb\" (UniqueName: \"kubernetes.io/projected/5c702aef-eccc-4da1-8040-c860c607a466-kube-api-access-p2vfb\") pod \"dns-default-99chm\" (UID: \"5c702aef-eccc-4da1-8040-c860c607a466\") " pod="openshift-dns/dns-default-99chm" Oct 30 21:28:26.260607 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.260589 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:26.273673 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.273647 2543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-htw5p"] Oct 30 21:28:26.341131 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.341093 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:28:26.341131 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.341098 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:28:26.341350 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.341138 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:28:26.341350 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.341093 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:28:26.341350 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.341124 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:28:26.341350 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.341177 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qcj4q\"" Oct 30 21:28:26.341350 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.341268 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2wf4q\"" Oct 30 21:28:26.341500 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.341354 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:28:26.359674 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.359645 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5c702aef-eccc-4da1-8040-c860c607a466-config-volume\") pod \"dns-default-99chm\" (UID: \"5c702aef-eccc-4da1-8040-c860c607a466\") " pod="openshift-dns/dns-default-99chm" Oct 30 21:28:26.359849 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.359678 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a31aa40d-e199-4095-b9b9-f2c7e31822e2-crio-socket\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:26.359849 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.359698 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a31aa40d-e199-4095-b9b9-f2c7e31822e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:26.359849 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.359799 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2vfb\" (UniqueName: \"kubernetes.io/projected/5c702aef-eccc-4da1-8040-c860c607a466-kube-api-access-p2vfb\") pod \"dns-default-99chm\" (UID: \"5c702aef-eccc-4da1-8040-c860c607a466\") " pod="openshift-dns/dns-default-99chm" Oct 30 21:28:26.359965 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.359892 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqfx8\" (UniqueName: \"kubernetes.io/projected/a31aa40d-e199-4095-b9b9-f2c7e31822e2-kube-api-access-kqfx8\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:26.360096 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.360065 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a31aa40d-e199-4095-b9b9-f2c7e31822e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:26.360127 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.360107 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c702aef-eccc-4da1-8040-c860c607a466-tmp-dir\") pod \"dns-default-99chm\" (UID: \"5c702aef-eccc-4da1-8040-c860c607a466\") " pod="openshift-dns/dns-default-99chm" Oct 30 21:28:26.360163 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.360149 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c702aef-eccc-4da1-8040-c860c607a466-metrics-tls\") pod \"dns-default-99chm\" (UID: \"5c702aef-eccc-4da1-8040-c860c607a466\") " pod="openshift-dns/dns-default-99chm" Oct 30 21:28:26.360198 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.360175 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ztrc\" (UniqueName: \"kubernetes.io/projected/808521e6-a0b3-4a24-b0b0-5a3cabffe0f5-kube-api-access-7ztrc\") pod \"ingress-canary-wdr8m\" (UID: \"808521e6-a0b3-4a24-b0b0-5a3cabffe0f5\") " pod="openshift-ingress-canary/ingress-canary-wdr8m" Oct 30 21:28:26.360234 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.360206 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/808521e6-a0b3-4a24-b0b0-5a3cabffe0f5-cert\") pod \"ingress-canary-wdr8m\" (UID: \"808521e6-a0b3-4a24-b0b0-5a3cabffe0f5\") " pod="openshift-ingress-canary/ingress-canary-wdr8m" Oct 30 21:28:26.360234 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.360224 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a31aa40d-e199-4095-b9b9-f2c7e31822e2-data-volume\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:26.360432 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.360410 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5c702aef-eccc-4da1-8040-c860c607a466-config-volume\") pod \"dns-default-99chm\" (UID: \"5c702aef-eccc-4da1-8040-c860c607a466\") " pod="openshift-dns/dns-default-99chm" Oct 30 21:28:26.360432 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.360421 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c702aef-eccc-4da1-8040-c860c607a466-tmp-dir\") pod \"dns-default-99chm\" (UID: \"5c702aef-eccc-4da1-8040-c860c607a466\") " pod="openshift-dns/dns-default-99chm" Oct 30 21:28:26.364819 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.364789 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c702aef-eccc-4da1-8040-c860c607a466-metrics-tls\") pod \"dns-default-99chm\" (UID: \"5c702aef-eccc-4da1-8040-c860c607a466\") " pod="openshift-dns/dns-default-99chm" Oct 30 21:28:26.364922 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.364828 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/808521e6-a0b3-4a24-b0b0-5a3cabffe0f5-cert\") pod \"ingress-canary-wdr8m\" (UID: \"808521e6-a0b3-4a24-b0b0-5a3cabffe0f5\") " pod="openshift-ingress-canary/ingress-canary-wdr8m" Oct 30 21:28:26.372156 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.372134 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2vfb\" (UniqueName: \"kubernetes.io/projected/5c702aef-eccc-4da1-8040-c860c607a466-kube-api-access-p2vfb\") pod \"dns-default-99chm\" (UID: \"5c702aef-eccc-4da1-8040-c860c607a466\") " pod="openshift-dns/dns-default-99chm" Oct 30 21:28:26.372341 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.372323 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ztrc\" (UniqueName: \"kubernetes.io/projected/808521e6-a0b3-4a24-b0b0-5a3cabffe0f5-kube-api-access-7ztrc\") pod \"ingress-canary-wdr8m\" (UID: \"808521e6-a0b3-4a24-b0b0-5a3cabffe0f5\") " pod="openshift-ingress-canary/ingress-canary-wdr8m" Oct 30 21:28:26.461533 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.461443 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a31aa40d-e199-4095-b9b9-f2c7e31822e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:26.461533 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.461496 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a31aa40d-e199-4095-b9b9-f2c7e31822e2-data-volume\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:26.461533 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.461526 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a31aa40d-e199-4095-b9b9-f2c7e31822e2-crio-socket\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:26.461732 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.461541 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a31aa40d-e199-4095-b9b9-f2c7e31822e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:26.461732 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.461566 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqfx8\" (UniqueName: \"kubernetes.io/projected/a31aa40d-e199-4095-b9b9-f2c7e31822e2-kube-api-access-kqfx8\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:26.461834 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.461780 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a31aa40d-e199-4095-b9b9-f2c7e31822e2-crio-socket\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:26.462033 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.461989 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a31aa40d-e199-4095-b9b9-f2c7e31822e2-data-volume\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:26.462141 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.462125 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a31aa40d-e199-4095-b9b9-f2c7e31822e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:26.463660 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.463643 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a31aa40d-e199-4095-b9b9-f2c7e31822e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:26.474035 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.474016 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqfx8\" (UniqueName: \"kubernetes.io/projected/a31aa40d-e199-4095-b9b9-f2c7e31822e2-kube-api-access-kqfx8\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:26.545085 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.545038 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wdr8m" Oct 30 21:28:26.553383 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.553362 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-99chm" Oct 30 21:28:26.575589 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.575556 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:26.682066 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.682035 2543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-54ddb8d6b-4rlwf"] Oct 30 21:28:26.686110 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.686089 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:26.694487 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.694463 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 21:28:26.694705 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.694688 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dj9gj\"" Oct 30 21:28:26.694780 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.694747 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 21:28:26.709118 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.709090 2543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54ddb8d6b-4rlwf"] Oct 30 21:28:26.763529 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.763449 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-service-ca\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:26.763529 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.763510 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-config\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:26.763719 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.763545 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-oauth-serving-cert\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:26.763719 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.763608 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-serving-cert\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:26.763719 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.763666 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-oauth-config\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:26.763719 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.763690 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl65r\" (UniqueName: \"kubernetes.io/projected/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-kube-api-access-sl65r\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:26.763872 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.763726 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-tmp\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:26.777035 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.777012 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 21:28:26.777142 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.777055 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 21:28:26.777142 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.777074 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 21:28:26.777142 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.777055 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 21:28:26.777957 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.777944 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 21:28:26.836659 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.836627 2543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wdr8m"] Oct 30 21:28:26.840929 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.840812 2543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-99chm"] Oct 30 21:28:26.842713 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:28:26.842684 2543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod808521e6_a0b3_4a24_b0b0_5a3cabffe0f5.slice/crio-593202543ae57c5e203fa3abb539eb711165e48b2bf2ba5cef1721e0bcc17287 WatchSource:0}: Error finding container 593202543ae57c5e203fa3abb539eb711165e48b2bf2ba5cef1721e0bcc17287: Status 404 returned error can't find the container with id 593202543ae57c5e203fa3abb539eb711165e48b2bf2ba5cef1721e0bcc17287 Oct 30 21:28:26.844694 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:28:26.844662 2543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c702aef_eccc_4da1_8040_c860c607a466.slice/crio-dfad36cb6e1ed10a6cd6660f40a16c252c832f5e86b10f65345c745eaaae034d WatchSource:0}: Error finding container dfad36cb6e1ed10a6cd6660f40a16c252c832f5e86b10f65345c745eaaae034d: Status 404 returned error can't find the container with id dfad36cb6e1ed10a6cd6660f40a16c252c832f5e86b10f65345c745eaaae034d Oct 30 21:28:26.864238 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.864214 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-serving-cert\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:26.864353 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.864254 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-oauth-config\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:26.864353 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.864273 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl65r\" (UniqueName: \"kubernetes.io/projected/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-kube-api-access-sl65r\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:26.864353 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.864309 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-tmp\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:26.864353 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.864335 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-service-ca\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:26.864551 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.864362 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-config\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:26.864551 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.864471 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-oauth-serving-cert\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:26.864718 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.864696 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-tmp\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:26.865097 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.865078 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-config\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:26.865189 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.865148 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-oauth-serving-cert\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:26.865250 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.865184 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-service-ca\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:26.866625 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.866598 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-serving-cert\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:26.866709 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.866640 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-oauth-config\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:26.872580 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.872561 2543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-htw5p"] Oct 30 21:28:26.875203 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:28:26.875181 2543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31aa40d_e199_4095_b9b9_f2c7e31822e2.slice/crio-828aaed34bc5f0464c0e592518791a70af36bb4a367070d703daf52da7d54d26 WatchSource:0}: Error finding container 828aaed34bc5f0464c0e592518791a70af36bb4a367070d703daf52da7d54d26: Status 404 returned error can't find the container with id 828aaed34bc5f0464c0e592518791a70af36bb4a367070d703daf52da7d54d26 Oct 30 21:28:26.878432 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.878415 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl65r\" (UniqueName: \"kubernetes.io/projected/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-kube-api-access-sl65r\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:26.982145 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.982112 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:28:26.982301 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.982153 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:28:26.987256 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.987229 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:28:26.987256 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.987242 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:28:26.987651 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.987512 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-f4rkh\"" Oct 30 21:28:26.987651 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.987577 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cjjj4\"" Oct 30 21:28:26.987651 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.987576 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:28:26.996919 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:26.996899 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:27.267769 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.267725 2543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54ddb8d6b-4rlwf"] Oct 30 21:28:27.270813 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:28:27.270791 2543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189b46e5_b36a_4ce4_be3e_b5486b7e45a8.slice/crio-a5510e59bc4d70b7949d7a6c736e5765e4da41eecb6bfbfed83b31498a7c5426 WatchSource:0}: Error finding container a5510e59bc4d70b7949d7a6c736e5765e4da41eecb6bfbfed83b31498a7c5426: Status 404 returned error can't find the container with id a5510e59bc4d70b7949d7a6c736e5765e4da41eecb6bfbfed83b31498a7c5426 Oct 30 21:28:27.315390 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.315354 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-htw5p" event={"ID":"a31aa40d-e199-4095-b9b9-f2c7e31822e2","Type":"ContainerStarted","Data":"4cef6c2deb646c4e467362946838be0e367548fcc67e4dddc433ba3d1910d4d2"} Oct 30 21:28:27.315576 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.315427 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-htw5p" event={"ID":"a31aa40d-e199-4095-b9b9-f2c7e31822e2","Type":"ContainerStarted","Data":"828aaed34bc5f0464c0e592518791a70af36bb4a367070d703daf52da7d54d26"} Oct 30 21:28:27.316868 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.316840 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-99chm" event={"ID":"5c702aef-eccc-4da1-8040-c860c607a466","Type":"ContainerStarted","Data":"dfad36cb6e1ed10a6cd6660f40a16c252c832f5e86b10f65345c745eaaae034d"} Oct 30 21:28:27.318395 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.318364 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54ddb8d6b-4rlwf" event={"ID":"189b46e5-b36a-4ce4-be3e-b5486b7e45a8","Type":"ContainerStarted","Data":"a5510e59bc4d70b7949d7a6c736e5765e4da41eecb6bfbfed83b31498a7c5426"} Oct 30 21:28:27.319607 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.319582 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wdr8m" event={"ID":"808521e6-a0b3-4a24-b0b0-5a3cabffe0f5","Type":"ContainerStarted","Data":"593202543ae57c5e203fa3abb539eb711165e48b2bf2ba5cef1721e0bcc17287"} Oct 30 21:28:27.781505 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.781473 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:27.807420 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.807386 2543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc"] Oct 30 21:28:27.810609 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.810585 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" Oct 30 21:28:27.814857 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.814327 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 21:28:27.815768 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.815731 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:28:27.815854 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.815794 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 21:28:27.815917 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.815858 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-75pfq\"" Oct 30 21:28:27.816556 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.816340 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:28:27.816767 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.816742 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:28:27.822837 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.822809 2543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc"] Oct 30 21:28:27.873489 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.873440 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f1829b1-8087-46b1-b64d-e3f15811caf6-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xh9wc\" (UID: \"7f1829b1-8087-46b1-b64d-e3f15811caf6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" Oct 30 21:28:27.873814 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.873554 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f1829b1-8087-46b1-b64d-e3f15811caf6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xh9wc\" (UID: \"7f1829b1-8087-46b1-b64d-e3f15811caf6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" Oct 30 21:28:27.873814 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.873627 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f1829b1-8087-46b1-b64d-e3f15811caf6-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xh9wc\" (UID: \"7f1829b1-8087-46b1-b64d-e3f15811caf6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" Oct 30 21:28:27.873814 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.873673 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z27h9\" (UniqueName: \"kubernetes.io/projected/7f1829b1-8087-46b1-b64d-e3f15811caf6-kube-api-access-z27h9\") pod \"prometheus-operator-9cd89b74c-xh9wc\" (UID: \"7f1829b1-8087-46b1-b64d-e3f15811caf6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" Oct 30 21:28:27.882348 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.882215 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:27.974357 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.974295 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f1829b1-8087-46b1-b64d-e3f15811caf6-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xh9wc\" (UID: \"7f1829b1-8087-46b1-b64d-e3f15811caf6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" Oct 30 21:28:27.974357 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.974362 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f1829b1-8087-46b1-b64d-e3f15811caf6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xh9wc\" (UID: \"7f1829b1-8087-46b1-b64d-e3f15811caf6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" Oct 30 21:28:27.974747 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.974426 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f1829b1-8087-46b1-b64d-e3f15811caf6-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xh9wc\" (UID: \"7f1829b1-8087-46b1-b64d-e3f15811caf6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" Oct 30 21:28:27.974747 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.974487 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z27h9\" (UniqueName: \"kubernetes.io/projected/7f1829b1-8087-46b1-b64d-e3f15811caf6-kube-api-access-z27h9\") pod \"prometheus-operator-9cd89b74c-xh9wc\" (UID: \"7f1829b1-8087-46b1-b64d-e3f15811caf6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" Oct 30 21:28:27.975554 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.975513 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f1829b1-8087-46b1-b64d-e3f15811caf6-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xh9wc\" (UID: \"7f1829b1-8087-46b1-b64d-e3f15811caf6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" Oct 30 21:28:27.981452 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.981422 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f1829b1-8087-46b1-b64d-e3f15811caf6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xh9wc\" (UID: \"7f1829b1-8087-46b1-b64d-e3f15811caf6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" Oct 30 21:28:27.984959 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.984198 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:27.988431 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.988405 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f1829b1-8087-46b1-b64d-e3f15811caf6-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xh9wc\" (UID: \"7f1829b1-8087-46b1-b64d-e3f15811caf6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" Oct 30 21:28:27.988937 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.988542 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rpdb2\"" Oct 30 21:28:27.988937 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.988901 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:28:27.990773 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.990366 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:27.993540 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:27.993397 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z27h9\" (UniqueName: \"kubernetes.io/projected/7f1829b1-8087-46b1-b64d-e3f15811caf6-kube-api-access-z27h9\") pod \"prometheus-operator-9cd89b74c-xh9wc\" (UID: \"7f1829b1-8087-46b1-b64d-e3f15811caf6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" Oct 30 21:28:28.132425 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:28.132350 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" Oct 30 21:28:28.152532 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:28.152463 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:28.326138 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:28.326099 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-htw5p" event={"ID":"a31aa40d-e199-4095-b9b9-f2c7e31822e2","Type":"ContainerStarted","Data":"97674ca9944ad16021734be45e5046d208105f5ae9f07f8e84594c4f60a46f54"} Oct 30 21:28:28.432748 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:28.432618 2543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc"] Oct 30 21:28:28.439349 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:28:28.439315 2543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f1829b1_8087_46b1_b64d_e3f15811caf6.slice/crio-d7b9f6a71d269f8ddf38e89179e4d4b9b8da09b774010f87c7d5614ddb1043f7 WatchSource:0}: Error finding container d7b9f6a71d269f8ddf38e89179e4d4b9b8da09b774010f87c7d5614ddb1043f7: Status 404 returned error can't find the container with id d7b9f6a71d269f8ddf38e89179e4d4b9b8da09b774010f87c7d5614ddb1043f7 Oct 30 21:28:28.874735 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:28.874653 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:28.989437 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:28.989354 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:29.333239 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:29.333199 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-99chm" event={"ID":"5c702aef-eccc-4da1-8040-c860c607a466","Type":"ContainerStarted","Data":"f856473da13812d3bb9bcad1ce70da4b3b87c6880886a56a1b92b46655323ca4"} Oct 30 21:28:29.333418 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:29.333280 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-99chm" event={"ID":"5c702aef-eccc-4da1-8040-c860c607a466","Type":"ContainerStarted","Data":"ace759483003c3dfd1b735547635847adfd7b49d28a24a50c05c9010481e11b3"} Oct 30 21:28:29.333418 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:29.333326 2543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-99chm" Oct 30 21:28:29.335618 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:29.335587 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" event={"ID":"7f1829b1-8087-46b1-b64d-e3f15811caf6","Type":"ContainerStarted","Data":"d7b9f6a71d269f8ddf38e89179e4d4b9b8da09b774010f87c7d5614ddb1043f7"} Oct 30 21:28:29.355519 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:29.355454 2543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-99chm" podStartSLOduration=1.232575277 podStartE2EDuration="3.355434041s" totalImagesPullingTime="2.122858764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:29.353444742 +0000 UTC m=+49.942822915" watchObservedRunningTime="2026-10-30 21:28:29.355434041 +0000 UTC m=+49.944812220" Oct 30 21:28:29.801193 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:29.801112 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:29.876006 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:29.875980 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-99chm_5c702aef-eccc-4da1-8040-c860c607a466/dns/0.log" Oct 30 21:28:29.901553 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:29.901522 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:30.105238 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:30.105212 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-99chm_5c702aef-eccc-4da1-8040-c860c607a466/kube-rbac-proxy/0.log" Oct 30 21:28:30.130307 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:30.130274 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:30.343444 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:30.343415 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-htw5p" event={"ID":"a31aa40d-e199-4095-b9b9-f2c7e31822e2","Type":"ContainerStarted","Data":"079da0906167c22af6f9e40b9d80a442a06088277a84811b21e97734ab9731ad"} Oct 30 21:28:30.348367 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:30.348194 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" event={"ID":"7f1829b1-8087-46b1-b64d-e3f15811caf6","Type":"ContainerStarted","Data":"5867718d080ad037bd6beb4a560419c88228bc1f76155564b79c9eaa4c61c355"} Oct 30 21:28:30.348367 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:30.348251 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" event={"ID":"7f1829b1-8087-46b1-b64d-e3f15811caf6","Type":"ContainerStarted","Data":"616d78783b36e13591eca55acffbbb79c228499cc372d647fce488437aa0900a"} Oct 30 21:28:30.370828 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:30.370780 2543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-htw5p" podStartSLOduration=1.7076685 podStartE2EDuration="4.370748011s" totalImagesPullingTime="2.663079511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:28:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:30.369956078 +0000 UTC m=+50.959334251" watchObservedRunningTime="2026-10-30 21:28:30.370748011 +0000 UTC m=+50.960126183" Oct 30 21:28:30.396695 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:30.396641 2543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" podStartSLOduration=1.777539161 podStartE2EDuration="3.396621558s" totalImagesPullingTime="1.619082397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:30.395808857 +0000 UTC m=+50.985187028" watchObservedRunningTime="2026-10-30 21:28:30.396621558 +0000 UTC m=+50.985999728" Oct 30 21:28:30.477246 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:30.477133 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:30.873721 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:30.873695 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j2j9f_c3af705e-b3ba-4379-8214-07ba1b431824/dns-node-resolver/0.log" Oct 30 21:28:31.352841 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:31.352807 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54ddb8d6b-4rlwf" event={"ID":"189b46e5-b36a-4ce4-be3e-b5486b7e45a8","Type":"ContainerStarted","Data":"d1335f87de32c5c790596fddc84b76e1f6055e69ce04d0c51156c382c0a9d20f"} Oct 30 21:28:31.354267 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:31.354238 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wdr8m" event={"ID":"808521e6-a0b3-4a24-b0b0-5a3cabffe0f5","Type":"ContainerStarted","Data":"f465bacbf8d50868edc5dcbe29b246789f7d0b5b95d4af77581e16e96e192ff4"} Oct 30 21:28:31.375421 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:31.375367 2543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-54ddb8d6b-4rlwf" podStartSLOduration=1.629749286 podStartE2EDuration="5.375353734s" totalImagesPullingTime="3.745604448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:31.374900736 +0000 UTC m=+51.964278920" watchObservedRunningTime="2026-10-30 21:28:31.375353734 +0000 UTC m=+51.964731905" Oct 30 21:28:31.397488 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:31.397428 2543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wdr8m" podStartSLOduration=1.929255583 podStartE2EDuration="5.397414811s" totalImagesPullingTime="3.468159228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:31.396186465 +0000 UTC m=+51.985564636" watchObservedRunningTime="2026-10-30 21:28:31.397414811 +0000 UTC m=+51.986792980" Oct 30 21:28:31.604879 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:31.604795 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:31.705146 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:31.705110 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:31.995084 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:31.995002 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:32.072877 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.072851 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-db7tz_5934c36a-2955-4f0c-a8e8-615f2fa07454/node-ca/0.log" Oct 30 21:28:32.205006 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.204967 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:32.232942 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.232903 2543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-h847d"] Oct 30 21:28:32.235988 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.235967 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.324543 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.324505 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21593b07-35ab-409e-98ef-74167147e27c-metrics-client-ca\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.324742 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.324554 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-accelerators-collector-config\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.324742 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.324635 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-tls\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.324742 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.324662 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz24x\" (UniqueName: \"kubernetes.io/projected/21593b07-35ab-409e-98ef-74167147e27c-kube-api-access-lz24x\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.324742 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.324699 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-textfile\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.324935 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.324818 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-wtmp\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.324935 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.324856 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21593b07-35ab-409e-98ef-74167147e27c-sys\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.324935 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.324879 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.324935 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.324910 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/21593b07-35ab-409e-98ef-74167147e27c-root\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.328452 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.328435 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9vdcb\"" Oct 30 21:28:32.328452 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.328433 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:28:32.328666 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.328649 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:28:32.330649 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.330625 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:32.333535 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.333521 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:28:32.426209 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.426167 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-tls\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.426685 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.426234 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz24x\" (UniqueName: \"kubernetes.io/projected/21593b07-35ab-409e-98ef-74167147e27c-kube-api-access-lz24x\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.426685 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.426274 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-textfile\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.426685 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.426333 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-wtmp\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.426685 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.426362 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21593b07-35ab-409e-98ef-74167147e27c-sys\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.426685 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.426409 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.426685 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.426454 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21593b07-35ab-409e-98ef-74167147e27c-sys\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.426685 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.426528 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-wtmp\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.426685 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.426664 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-textfile\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.427085 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.426715 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/21593b07-35ab-409e-98ef-74167147e27c-root\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.427085 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.426746 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21593b07-35ab-409e-98ef-74167147e27c-metrics-client-ca\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.427085 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.426809 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/21593b07-35ab-409e-98ef-74167147e27c-root\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.427085 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.426814 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-accelerators-collector-config\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.427276 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.427236 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-accelerators-collector-config\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.427276 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.427268 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21593b07-35ab-409e-98ef-74167147e27c-metrics-client-ca\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.428629 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.428608 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.428800 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.428785 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-tls\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.432806 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.432787 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:32.438475 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.438457 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz24x\" (UniqueName: \"kubernetes.io/projected/21593b07-35ab-409e-98ef-74167147e27c-kube-api-access-lz24x\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.538189 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.538158 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:32.552302 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.552232 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:32.563331 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:28:32.563302 2543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21593b07_35ab_409e_98ef_74167147e27c.slice/crio-04996af222f21f33acf2dbb3b6b94b8be5ca45bb4627d0208b7720a427210892 WatchSource:0}: Error finding container 04996af222f21f33acf2dbb3b6b94b8be5ca45bb4627d0208b7720a427210892: Status 404 returned error can't find the container with id 04996af222f21f33acf2dbb3b6b94b8be5ca45bb4627d0208b7720a427210892 Oct 30 21:28:32.639394 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.639299 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:32.741436 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.741396 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:32.840844 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.840812 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:32.874828 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:32.874805 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wdr8m_808521e6-a0b3-4a24-b0b0-5a3cabffe0f5/serve-healthcheck-canary/0.log" Oct 30 21:28:33.252031 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:33.251998 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:33.274088 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:33.274060 2543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:33.361864 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:33.361825 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h847d" event={"ID":"21593b07-35ab-409e-98ef-74167147e27c","Type":"ContainerStarted","Data":"04996af222f21f33acf2dbb3b6b94b8be5ca45bb4627d0208b7720a427210892"} Oct 30 21:28:33.382155 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:33.382122 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:33.497380 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:33.497342 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:33.970152 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:33.970071 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:34.144907 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:34.144878 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:34.249410 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:34.249329 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:34.369765 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:34.369714 2543 generic.go:356] "Generic (PLEG): container finished" podID="21593b07-35ab-409e-98ef-74167147e27c" containerID="c091ea83c74aad3ea2fe30942815804a025d15ae29391539061b22d15600e580" exitCode=0 Oct 30 21:28:34.369921 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:34.369798 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h847d" event={"ID":"21593b07-35ab-409e-98ef-74167147e27c","Type":"ContainerDied","Data":"c091ea83c74aad3ea2fe30942815804a025d15ae29391539061b22d15600e580"} Oct 30 21:28:35.336241 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:35.336207 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:35.374785 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:35.374736 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h847d" event={"ID":"21593b07-35ab-409e-98ef-74167147e27c","Type":"ContainerStarted","Data":"18cf36c5a6839d4c679cd45c5fd671e0974545ac14472229a6ac9c15ebbece28"} Oct 30 21:28:35.374941 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:35.374804 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h847d" event={"ID":"21593b07-35ab-409e-98ef-74167147e27c","Type":"ContainerStarted","Data":"7ee7340df160b1da22ab94bb47e88f6fe1c0793994a954dbc9da7f7b9c776572"} Oct 30 21:28:35.399916 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:35.399865 2543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-h847d" podStartSLOduration=2.317287515 podStartE2EDuration="3.399853753s" totalImagesPullingTime="1.017375337s" totalInitContainerRuntime="65.190901ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:35.398619987 +0000 UTC m=+55.987998199" watchObservedRunningTime="2026-10-30 21:28:35.399853753 +0000 UTC m=+55.989231946" Oct 30 21:28:35.444944 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:35.444914 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:35.552908 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:35.552877 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:35.680679 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:35.680604 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:35.775354 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:35.775324 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:36.998046 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:36.998012 2543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:36.998487 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:36.998073 2543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:37.002950 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.002925 2543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:37.057496 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.057464 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:37.167401 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.167369 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:37.384337 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.384309 2543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:37.473777 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.473731 2543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6d84c6fb95-spv7p"] Oct 30 21:28:37.477068 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.477048 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.481280 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.481257 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 21:28:37.481403 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.481367 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 21:28:37.482476 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.482455 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 21:28:37.482605 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.482475 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 21:28:37.482853 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.482837 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 21:28:37.484267 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.484251 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-qhqpw\"" Oct 30 21:28:37.489947 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.489924 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 21:28:37.494793 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.494745 2543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6d84c6fb95-spv7p"] Oct 30 21:28:37.566460 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.566422 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5c17e9-1229-4d33-ba12-3eab4727e64d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.566460 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.566465 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da5c17e9-1229-4d33-ba12-3eab4727e64d-metrics-client-ca\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.566666 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.566500 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/da5c17e9-1229-4d33-ba12-3eab4727e64d-federate-client-tls\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.566666 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.566592 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da5c17e9-1229-4d33-ba12-3eab4727e64d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.566666 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.566613 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/da5c17e9-1229-4d33-ba12-3eab4727e64d-secret-telemeter-client\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.566666 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.566629 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5c17e9-1229-4d33-ba12-3eab4727e64d-serving-certs-ca-bundle\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.566666 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.566648 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmtxv\" (UniqueName: \"kubernetes.io/projected/da5c17e9-1229-4d33-ba12-3eab4727e64d-kube-api-access-rmtxv\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.566887 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.566681 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/da5c17e9-1229-4d33-ba12-3eab4727e64d-telemeter-client-tls\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.667538 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.667459 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/da5c17e9-1229-4d33-ba12-3eab4727e64d-secret-telemeter-client\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.667538 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.667499 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5c17e9-1229-4d33-ba12-3eab4727e64d-serving-certs-ca-bundle\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.667538 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.667517 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmtxv\" (UniqueName: \"kubernetes.io/projected/da5c17e9-1229-4d33-ba12-3eab4727e64d-kube-api-access-rmtxv\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.667538 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.667538 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/da5c17e9-1229-4d33-ba12-3eab4727e64d-telemeter-client-tls\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.667894 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.667572 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5c17e9-1229-4d33-ba12-3eab4727e64d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.667894 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.667590 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da5c17e9-1229-4d33-ba12-3eab4727e64d-metrics-client-ca\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.667894 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.667631 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/da5c17e9-1229-4d33-ba12-3eab4727e64d-federate-client-tls\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.667894 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.667715 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da5c17e9-1229-4d33-ba12-3eab4727e64d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.668466 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.668441 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5c17e9-1229-4d33-ba12-3eab4727e64d-serving-certs-ca-bundle\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.668721 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.668697 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5c17e9-1229-4d33-ba12-3eab4727e64d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.668835 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.668697 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da5c17e9-1229-4d33-ba12-3eab4727e64d-metrics-client-ca\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.670206 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.670178 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/da5c17e9-1229-4d33-ba12-3eab4727e64d-telemeter-client-tls\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.670325 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.670303 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/da5c17e9-1229-4d33-ba12-3eab4727e64d-federate-client-tls\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.670530 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.670512 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/da5c17e9-1229-4d33-ba12-3eab4727e64d-secret-telemeter-client\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.670599 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.670578 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da5c17e9-1229-4d33-ba12-3eab4727e64d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.684561 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.684541 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmtxv\" (UniqueName: \"kubernetes.io/projected/da5c17e9-1229-4d33-ba12-3eab4727e64d-kube-api-access-rmtxv\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:37.795035 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:37.795001 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:38.082438 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.082288 2543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6d84c6fb95-spv7p"] Oct 30 21:28:38.086526 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:28:38.086500 2543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5c17e9_1229_4d33_ba12_3eab4727e64d.slice/crio-4067ec29eb4ea5cbff18adbbd3d2f6346dfa9ffdc48fbb0744ac13476c2fb181 WatchSource:0}: Error finding container 4067ec29eb4ea5cbff18adbbd3d2f6346dfa9ffdc48fbb0744ac13476c2fb181: Status 404 returned error can't find the container with id 4067ec29eb4ea5cbff18adbbd3d2f6346dfa9ffdc48fbb0744ac13476c2fb181 Oct 30 21:28:38.384644 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.384559 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" event={"ID":"da5c17e9-1229-4d33-ba12-3eab4727e64d","Type":"ContainerStarted","Data":"4067ec29eb4ea5cbff18adbbd3d2f6346dfa9ffdc48fbb0744ac13476c2fb181"} Oct 30 21:28:38.638612 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.638508 2543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:28:38.642491 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.642468 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.651584 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.651504 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 21:28:38.651703 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.651687 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 21:28:38.651782 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.651728 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 21:28:38.658070 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.656940 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 21:28:38.658070 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.657002 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 21:28:38.658070 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.657778 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 21:28:38.660159 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.660084 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 21:28:38.660310 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.660209 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 21:28:38.660310 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.660284 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 21:28:38.662519 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.662312 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 21:28:38.662519 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.662489 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 21:28:38.671274 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.671230 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vf75f\"" Oct 30 21:28:38.671398 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.671301 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 21:28:38.671398 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.671325 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 21:28:38.671579 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.671552 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7mnf0orvh0m0j\"" Oct 30 21:28:38.671579 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.671554 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 21:28:38.672862 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.672836 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:38.674117 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.674027 2543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:28:38.677057 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.677036 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.677162 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.677069 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.677162 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.677097 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.677162 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.677121 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.677162 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.677156 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.677365 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.677204 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.677365 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.677228 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.677365 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.677343 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-config-out\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.677467 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.677421 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.677508 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.677467 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.677582 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.677565 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.677643 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.677618 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.677696 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.677669 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.677769 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.677698 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ppdt\" (UniqueName: \"kubernetes.io/projected/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-kube-api-access-4ppdt\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.678019 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.677762 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.678019 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.677791 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-config\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.678019 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.677817 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.678019 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.677842 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.678019 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.677868 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.678019 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.677896 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-web-config\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.753205 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.753172 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 21:28:38.779915 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.779882 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.779915 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.779919 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.780162 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.779947 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-web-config\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.780162 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.779989 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.780162 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.780015 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.783804 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.780572 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.783804 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.780618 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.783804 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.780668 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.783804 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.781097 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.783804 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.781144 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.783804 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.781323 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-config-out\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.783804 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.781366 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.783804 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.781397 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.783804 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.781438 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.783804 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.781464 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.783804 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.781500 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.783804 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.781530 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ppdt\" (UniqueName: \"kubernetes.io/projected/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-kube-api-access-4ppdt\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.783804 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.781558 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.783804 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.781590 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-config\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.783804 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.781616 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.783804 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.781918 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.783804 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.782155 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.784722 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.782676 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.784722 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.783166 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.784722 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.783411 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.784722 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.783485 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.784722 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.783592 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.784722 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.784082 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.786644 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.786552 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.786741 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.786662 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.787287 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.786826 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-web-config\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.787287 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.787223 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-config\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.787707 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.787683 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-config-out\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.787891 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.787873 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.788221 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.788205 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.788259 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.788227 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.788870 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.788824 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.789242 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.789217 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.789588 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.789566 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.800822 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.800798 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ppdt\" (UniqueName: \"kubernetes.io/projected/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-kube-api-access-4ppdt\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:38.977645 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:38.977570 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:39.290800 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:39.288969 2543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:28:39.350327 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:39.350303 2543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-99chm" Oct 30 21:28:39.390523 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:39.390493 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"999c5bdd-e3fc-4103-9d65-0d3a228fc78b","Type":"ContainerStarted","Data":"253215521b57eb57279a0ac998f8cbcd172f5d06f7d99cad8392adf60220daeb"} Oct 30 21:28:40.396097 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:40.396063 2543 generic.go:356] "Generic (PLEG): container finished" podID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerID="ef6880b249d4139651e75607e731fc966d7a838c977325ba312cb6be6292545c" exitCode=0 Oct 30 21:28:40.396433 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:40.396402 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"999c5bdd-e3fc-4103-9d65-0d3a228fc78b","Type":"ContainerDied","Data":"ef6880b249d4139651e75607e731fc966d7a838c977325ba312cb6be6292545c"} Oct 30 21:28:40.398194 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:40.398165 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" event={"ID":"da5c17e9-1229-4d33-ba12-3eab4727e64d","Type":"ContainerStarted","Data":"cd0d57795e13e4fcb7eb8a7ef8f20c8d0fe400f1504e064d499d1962533745d3"} Oct 30 21:28:41.403961 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:41.403920 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" event={"ID":"da5c17e9-1229-4d33-ba12-3eab4727e64d","Type":"ContainerStarted","Data":"fa0358fe203250fa397959ab0848f8d6afd60677e0d159ad044d3be75e29755d"} Oct 30 21:28:41.404354 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:41.403991 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" event={"ID":"da5c17e9-1229-4d33-ba12-3eab4727e64d","Type":"ContainerStarted","Data":"2947dd17afa28e9b1ec0046b7081d125e80ac72f87fdbf86a33cf664cacdbc77"} Oct 30 21:28:41.433345 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:41.433291 2543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" podStartSLOduration=2.069262225 podStartE2EDuration="4.433274467s" totalImagesPullingTime="2.364012242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:41.432502074 +0000 UTC m=+62.021880246" watchObservedRunningTime="2026-10-30 21:28:41.433274467 +0000 UTC m=+62.022652640" Oct 30 21:28:41.902973 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:41.902935 2543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c5fdfbf9b-l7clr"] Oct 30 21:28:41.906341 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:41.906321 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:41.917166 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:41.917143 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 21:28:41.919106 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:41.919084 2543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c5fdfbf9b-l7clr"] Oct 30 21:28:42.012586 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.012557 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d43f954-119d-40c5-9727-1e25043da4ee-console-oauth-config\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.012586 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.012598 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-trusted-ca-bundle\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.012828 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.012624 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bczbc\" (UniqueName: \"kubernetes.io/projected/8d43f954-119d-40c5-9727-1e25043da4ee-kube-api-access-bczbc\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.012828 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.012761 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d43f954-119d-40c5-9727-1e25043da4ee-tmp\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.012828 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.012816 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d43f954-119d-40c5-9727-1e25043da4ee-console-serving-cert\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.012993 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.012840 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-oauth-serving-cert\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.012993 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.012893 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-console-config\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.012993 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.012979 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-service-ca\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.113843 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.113815 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d43f954-119d-40c5-9727-1e25043da4ee-console-serving-cert\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.114004 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.113853 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-oauth-serving-cert\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.114004 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.113886 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-console-config\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.114125 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.114070 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-service-ca\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.114177 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.114147 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d43f954-119d-40c5-9727-1e25043da4ee-console-oauth-config\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.114228 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.114188 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-trusted-ca-bundle\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.114228 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.114213 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bczbc\" (UniqueName: \"kubernetes.io/projected/8d43f954-119d-40c5-9727-1e25043da4ee-kube-api-access-bczbc\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.114327 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.114251 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d43f954-119d-40c5-9727-1e25043da4ee-tmp\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.115142 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.114586 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-console-config\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.115142 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.114699 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d43f954-119d-40c5-9727-1e25043da4ee-tmp\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.115142 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.114714 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-oauth-serving-cert\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.115142 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.115070 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-service-ca\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.115142 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.115107 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-trusted-ca-bundle\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.116602 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.116578 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d43f954-119d-40c5-9727-1e25043da4ee-console-serving-cert\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.116705 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.116625 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d43f954-119d-40c5-9727-1e25043da4ee-console-oauth-config\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.127546 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.127524 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bczbc\" (UniqueName: \"kubernetes.io/projected/8d43f954-119d-40c5-9727-1e25043da4ee-kube-api-access-bczbc\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.219643 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.219547 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:42.507640 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:42.507611 2543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c5fdfbf9b-l7clr"] Oct 30 21:28:42.508100 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:28:42.508065 2543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d43f954_119d_40c5_9727_1e25043da4ee.slice/crio-bcd8b205dfbf9b7c1e054114904f54e7451342bcfa11344e056753f200b7694b WatchSource:0}: Error finding container bcd8b205dfbf9b7c1e054114904f54e7451342bcfa11344e056753f200b7694b: Status 404 returned error can't find the container with id bcd8b205dfbf9b7c1e054114904f54e7451342bcfa11344e056753f200b7694b Oct 30 21:28:43.139872 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:43.139829 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:43.239470 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:43.239435 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:43.373070 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:43.373032 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:43.412199 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:43.412172 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5fdfbf9b-l7clr" event={"ID":"8d43f954-119d-40c5-9727-1e25043da4ee","Type":"ContainerStarted","Data":"d30ed4895ec7878e057cb57171fae40237e2a25e793c73814789d39ed11a5de7"} Oct 30 21:28:43.412281 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:43.412222 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5fdfbf9b-l7clr" event={"ID":"8d43f954-119d-40c5-9727-1e25043da4ee","Type":"ContainerStarted","Data":"bcd8b205dfbf9b7c1e054114904f54e7451342bcfa11344e056753f200b7694b"} Oct 30 21:28:43.436273 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:43.434917 2543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5c5fdfbf9b-l7clr" podStartSLOduration=2.434899409 podStartE2EDuration="2.434899409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:43.434873572 +0000 UTC m=+64.024251754" watchObservedRunningTime="2026-10-30 21:28:43.434899409 +0000 UTC m=+64.024277581" Oct 30 21:28:43.477269 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:43.477245 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:44.339358 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:44.339317 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:44.418199 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:44.418162 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"999c5bdd-e3fc-4103-9d65-0d3a228fc78b","Type":"ContainerStarted","Data":"0ccb9b005019e336e5c868e7111c6f5a0c67848f0425b23b2f41ff4a5efd58fe"} Oct 30 21:28:44.418375 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:44.418252 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"999c5bdd-e3fc-4103-9d65-0d3a228fc78b","Type":"ContainerStarted","Data":"2a248c86f433a34a28abbdd7cf5827290268d64646a26957352516bb4d73ddf2"} Oct 30 21:28:44.860578 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:44.860532 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:44.967961 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:44.967915 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:45.648485 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:45.648448 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqcwl\" (UniqueName: \"kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl\") pod \"network-check-target-kv4cs\" (UID: \"bb88c1ff-4b28-4812-9b2b-8876983720e5\") " pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:28:45.651916 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:45.651898 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:28:45.661804 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:45.661786 2543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:28:45.671961 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:45.671934 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqcwl\" (UniqueName: \"kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl\") pod \"network-check-target-kv4cs\" (UID: \"bb88c1ff-4b28-4812-9b2b-8876983720e5\") " pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:28:45.749838 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:45.749806 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs\") pod \"network-metrics-daemon-4r4zb\" (UID: \"3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78\") " pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:28:45.749943 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:45.749861 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret\") pod \"global-pull-secret-syncer-c2vrg\" (UID: \"183318eb-952e-4e93-b716-6ab50b9c58dd\") " pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:45.753335 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:45.753314 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:28:45.754624 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:45.754608 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:28:45.762729 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:45.762709 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs\") pod \"network-metrics-daemon-4r4zb\" (UID: \"3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78\") " pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:28:45.763092 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:45.763004 2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret\") pod \"global-pull-secret-syncer-c2vrg\" (UID: \"183318eb-952e-4e93-b716-6ab50b9c58dd\") " pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:45.899030 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:45.898940 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cjjj4\"" Oct 30 21:28:45.901622 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:45.901598 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-f4rkh\"" Oct 30 21:28:45.906973 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:45.906954 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:28:45.909484 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:45.909420 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:28:46.015098 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:46.015074 2543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rpdb2\"" Oct 30 21:28:46.023129 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:46.023107 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:46.201157 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:46.201010 2543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4r4zb"] Oct 30 21:28:46.204002 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:28:46.203979 2543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8eebdf_2dfa_4039_8bb1_ca648e0a8d78.slice/crio-54fd1c126cabbff9f892111cc0bbff2664cb51ea952375ab1eb54b8401f6cbf8 WatchSource:0}: Error finding container 54fd1c126cabbff9f892111cc0bbff2664cb51ea952375ab1eb54b8401f6cbf8: Status 404 returned error can't find the container with id 54fd1c126cabbff9f892111cc0bbff2664cb51ea952375ab1eb54b8401f6cbf8 Oct 30 21:28:46.204178 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:46.204157 2543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kv4cs"] Oct 30 21:28:46.205658 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:28:46.205638 2543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb88c1ff_4b28_4812_9b2b_8876983720e5.slice/crio-6c95fd8aec2140a47abfc68a4ebcb6b3b50eec9e9579bc892c31a5fff26eb8e7 WatchSource:0}: Error finding container 6c95fd8aec2140a47abfc68a4ebcb6b3b50eec9e9579bc892c31a5fff26eb8e7: Status 404 returned error can't find the container with id 6c95fd8aec2140a47abfc68a4ebcb6b3b50eec9e9579bc892c31a5fff26eb8e7 Oct 30 21:28:46.293626 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:46.293594 2543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c2vrg"] Oct 30 21:28:46.296953 ip-10-0-140-163 kubenswrapper[2543]: W1030 21:28:46.296926 2543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod183318eb_952e_4e93_b716_6ab50b9c58dd.slice/crio-aafb99e79b91dc43038e1c3d53e6955437495733c43e838913a87c6b2f17a553 WatchSource:0}: Error finding container aafb99e79b91dc43038e1c3d53e6955437495733c43e838913a87c6b2f17a553: Status 404 returned error can't find the container with id aafb99e79b91dc43038e1c3d53e6955437495733c43e838913a87c6b2f17a553 Oct 30 21:28:46.435690 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:46.435650 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"999c5bdd-e3fc-4103-9d65-0d3a228fc78b","Type":"ContainerStarted","Data":"13a723438766d7bad387bff6a969b504d00ed2613d5b714084603e5a5842599f"} Oct 30 21:28:46.435883 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:46.435735 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"999c5bdd-e3fc-4103-9d65-0d3a228fc78b","Type":"ContainerStarted","Data":"923bee7254814f53592fe25939cea6f23dc928ef5cd2b2a84c8f28cbf46bdbc8"} Oct 30 21:28:46.435883 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:46.435803 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"999c5bdd-e3fc-4103-9d65-0d3a228fc78b","Type":"ContainerStarted","Data":"ca6b1295e7c5546f7289353bb4dc679654a0dafa29957bfa0a98be9ae2a9c130"} Oct 30 21:28:46.435883 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:46.435833 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"999c5bdd-e3fc-4103-9d65-0d3a228fc78b","Type":"ContainerStarted","Data":"bfb255833c61bfe2ed287a07c34622eaff10f9eea540a4764fedbce5957d847d"} Oct 30 21:28:46.437126 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:46.437082 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4r4zb" event={"ID":"3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78","Type":"ContainerStarted","Data":"54fd1c126cabbff9f892111cc0bbff2664cb51ea952375ab1eb54b8401f6cbf8"} Oct 30 21:28:46.438420 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:46.438395 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kv4cs" event={"ID":"bb88c1ff-4b28-4812-9b2b-8876983720e5","Type":"ContainerStarted","Data":"6c95fd8aec2140a47abfc68a4ebcb6b3b50eec9e9579bc892c31a5fff26eb8e7"} Oct 30 21:28:46.439745 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:46.439723 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c2vrg" event={"ID":"183318eb-952e-4e93-b716-6ab50b9c58dd","Type":"ContainerStarted","Data":"aafb99e79b91dc43038e1c3d53e6955437495733c43e838913a87c6b2f17a553"} Oct 30 21:28:46.488438 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:46.488333 2543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.344176225 podStartE2EDuration="8.488310887s" totalImagesPullingTime="6.124228809s" totalInitContainerRuntime="19.905853ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:46.484971877 +0000 UTC m=+67.074350062" watchObservedRunningTime="2026-10-30 21:28:46.488310887 +0000 UTC m=+67.077689059" Oct 30 21:28:47.490150 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:47.490088 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:47.609401 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:47.609374 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:47.723651 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:47.723623 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:47.832000 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:47.831970 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:48.451448 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:48.450828 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4r4zb" event={"ID":"3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78","Type":"ContainerStarted","Data":"6be642e59232d67dace11ebedebe7c14b53b2269ae3e2a0de97180f25164d39c"} Oct 30 21:28:48.451448 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:48.450897 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4r4zb" event={"ID":"3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78","Type":"ContainerStarted","Data":"613eb0e0bca941a15475009002eb0a1e13619ec14ce692adfac77816e5631a99"} Oct 30 21:28:48.475255 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:48.475193 2543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4r4zb" podStartSLOduration=67.082304885 podStartE2EDuration="1m8.475175476s" totalImagesPullingTime="1.392870591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:48.47376999 +0000 UTC m=+69.063148161" watchObservedRunningTime="2026-10-30 21:28:48.475175476 +0000 UTC m=+69.064553648" Oct 30 21:28:48.545307 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:48.545261 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:48.667036 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:48.666999 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:48.978705 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:48.978671 2543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:50.464882 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:50.464842 2543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kv4cs" event={"ID":"bb88c1ff-4b28-4812-9b2b-8876983720e5","Type":"ContainerStarted","Data":"0bb935d7a7f9c8789d7d44786bb1d58159f367678549dd0aa8c3b673adc000aa"} Oct 30 21:28:50.465282 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:50.465029 2543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:28:50.486779 ip-10-0-140-163 kubenswrapper[2543]: I1030 21:28:50.486690 2543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kv4cs" podStartSLOduration=67.98109766 podStartE2EDuration="1m11.486672706s" totalImagesPullingTime="3.505575046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:50.484602745 +0000 UTC m=+71.073980918" watchObservedRunningTime="2026-10-30 21:28:50.486672706 +0000 UTC m=+71.076050875" Oct 30 21:28:51.344635 ip-10-0-140-163 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 21:28:51.358131 ip-10-0-140-163 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 21:28:51.358294 ip-10-0-140-163 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 21:28:51.358343 ip-10-0-140-163 systemd[1]: kubelet.service: Consumed 4.112s CPU time. Oct 30 21:28:51.371447 ip-10-0-140-163 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:28:51.432931 ip-10-0-140-163 kubenswrapper[5801]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:28:51.432931 ip-10-0-140-163 kubenswrapper[5801]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:28:51.432931 ip-10-0-140-163 kubenswrapper[5801]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:28:51.432931 ip-10-0-140-163 kubenswrapper[5801]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:28:51.434947 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.434862 5801 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:28:51.434947 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.434947 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:28:51.435021 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.434952 5801 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:28:51.435021 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.434955 5801 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:28:51.435021 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.434958 5801 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:28:51.435021 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.434961 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:28:51.435021 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.434964 5801 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:28:51.435021 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.434967 5801 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:28:51.435021 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.434970 5801 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:28:51.435021 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.434973 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:28:51.435021 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.434976 5801 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:28:51.435021 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.434978 5801 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:28:51.435021 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.434981 5801 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:28:51.435021 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.434984 5801 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:28:51.435021 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.434987 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:28:51.435021 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.434989 5801 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:28:51.435021 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.434992 5801 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:28:51.435021 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.434995 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:28:51.435021 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.434998 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:28:51.435021 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435000 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:28:51.435021 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435004 5801 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:28:51.435481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435007 5801 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:28:51.435481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435010 5801 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:28:51.435481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435012 5801 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:28:51.435481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435015 5801 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:28:51.435481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435018 5801 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:28:51.435481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435020 5801 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:28:51.435481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435023 5801 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:28:51.435481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435026 5801 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:28:51.435481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435029 5801 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:28:51.435481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435032 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:28:51.435481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435035 5801 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:28:51.435481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435037 5801 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:28:51.435481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435040 5801 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:28:51.435481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435043 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:28:51.435481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435045 5801 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:28:51.435481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435048 5801 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:28:51.435481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435050 5801 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:28:51.435481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435053 5801 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:28:51.435481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435055 5801 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:28:51.435481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435058 5801 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:28:51.435979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435060 5801 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:28:51.435979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435063 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:28:51.435979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435065 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:28:51.435979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435068 5801 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:28:51.435979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435071 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:28:51.435979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435074 5801 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:28:51.435979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435076 5801 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:28:51.435979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435079 5801 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:28:51.435979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435081 5801 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:28:51.435979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435084 5801 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:28:51.435979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435088 5801 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:28:51.435979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435090 5801 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:28:51.435979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435093 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:28:51.435979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435097 5801 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:28:51.435979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435100 5801 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:28:51.435979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435103 5801 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:28:51.435979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435105 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:28:51.435979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435108 5801 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:28:51.435979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435111 5801 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:28:51.436449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435114 5801 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:28:51.436449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435118 5801 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:28:51.436449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435122 5801 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:28:51.436449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435125 5801 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:28:51.436449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435128 5801 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:28:51.436449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435130 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:28:51.436449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435133 5801 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:28:51.436449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435136 5801 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:28:51.436449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435139 5801 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:28:51.436449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435141 5801 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:28:51.436449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435144 5801 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:28:51.436449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435147 5801 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:28:51.436449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435150 5801 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:28:51.436449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435153 5801 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:28:51.436449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435155 5801 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:28:51.436449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435158 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:28:51.436449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435160 5801 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:28:51.436449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435163 5801 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:28:51.436449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435166 5801 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:28:51.436952 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435168 5801 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:28:51.436952 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435171 5801 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:28:51.436952 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435173 5801 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:28:51.436952 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435176 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:28:51.436952 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435178 5801 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:28:51.436952 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435181 5801 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:28:51.436952 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435183 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:28:51.436952 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435186 5801 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:28:51.436952 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435189 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:28:51.436952 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435191 5801 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:28:51.436952 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435194 5801 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:28:51.436952 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435196 5801 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:28:51.436952 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435200 5801 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:28:51.436952 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435203 5801 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:28:51.436952 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435206 5801 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:28:51.436952 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435208 5801 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:28:51.436952 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435211 5801 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:28:51.436952 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435214 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:28:51.436952 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435216 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:28:51.436952 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435220 5801 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:28:51.437453 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435223 5801 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:28:51.437453 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435225 5801 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:28:51.437453 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435228 5801 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:28:51.437453 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435230 5801 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:28:51.437453 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435233 5801 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:28:51.437453 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435235 5801 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:28:51.437453 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.435238 5801 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:28:51.437453 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436012 5801 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:28:51.437453 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436020 5801 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:28:51.437453 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436023 5801 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:28:51.437453 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436026 5801 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:28:51.437453 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436029 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:28:51.437453 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436032 5801 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:28:51.437453 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436035 5801 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:28:51.437453 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436038 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:28:51.437453 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436041 5801 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:28:51.437453 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436043 5801 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:28:51.437453 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436046 5801 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:28:51.437453 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436049 5801 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:28:51.437453 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436052 5801 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:28:51.438000 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436055 5801 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:28:51.438000 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436058 5801 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:28:51.438000 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436061 5801 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:28:51.438000 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436063 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:28:51.438000 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436067 5801 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:28:51.438000 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436070 5801 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:28:51.438000 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436072 5801 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:28:51.438000 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436075 5801 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:28:51.438000 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436078 5801 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:28:51.438000 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436081 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:28:51.438000 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436083 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:28:51.438000 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436086 5801 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:28:51.438000 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436088 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:28:51.438000 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436091 5801 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:28:51.438000 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436094 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:28:51.438000 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436096 5801 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:28:51.438000 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436099 5801 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:28:51.438000 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436102 5801 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:28:51.438000 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436104 5801 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:28:51.438481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436107 5801 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:28:51.438481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436110 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:28:51.438481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436112 5801 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:28:51.438481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436115 5801 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:28:51.438481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436117 5801 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:28:51.438481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436120 5801 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:28:51.438481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436123 5801 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:28:51.438481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436126 5801 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:28:51.438481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436128 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:28:51.438481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436131 5801 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:28:51.438481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436134 5801 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:28:51.438481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436137 5801 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:28:51.438481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436139 5801 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:28:51.438481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436142 5801 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:28:51.438481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436145 5801 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:28:51.438481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436147 5801 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:28:51.438481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436150 5801 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:28:51.438481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436153 5801 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:28:51.438481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436156 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:28:51.438481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436159 5801 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:28:51.439005 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436162 5801 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:28:51.439005 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436165 5801 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:28:51.439005 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436167 5801 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:28:51.439005 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436170 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:28:51.439005 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436172 5801 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:28:51.439005 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436175 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:28:51.439005 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436177 5801 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:28:51.439005 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436180 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:28:51.439005 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436183 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:28:51.439005 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436185 5801 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:28:51.439005 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436188 5801 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:28:51.439005 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436190 5801 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:28:51.439005 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436193 5801 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:28:51.439005 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436196 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:28:51.439005 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436199 5801 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:28:51.439005 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436203 5801 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:28:51.439005 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436207 5801 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:28:51.439005 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436210 5801 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:28:51.439005 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436213 5801 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:28:51.439500 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436215 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:28:51.439500 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436218 5801 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:28:51.439500 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436222 5801 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:28:51.439500 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436225 5801 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:28:51.439500 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436228 5801 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:28:51.439500 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436230 5801 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:28:51.439500 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436233 5801 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:28:51.439500 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436235 5801 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:28:51.439500 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436238 5801 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:28:51.439500 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436241 5801 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:28:51.439500 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436243 5801 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:28:51.439500 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436246 5801 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:28:51.439500 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436248 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:28:51.439500 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436251 5801 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:28:51.439500 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436254 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:28:51.439500 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436257 5801 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:28:51.439500 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436261 5801 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:28:51.439500 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436265 5801 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:28:51.439500 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436268 5801 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:28:51.439979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436271 5801 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:28:51.439979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436273 5801 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:28:51.439979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436276 5801 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:28:51.439979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436278 5801 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:28:51.439979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436281 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:28:51.439979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436284 5801 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:28:51.439979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436286 5801 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:28:51.439979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436289 5801 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:28:51.439979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436292 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:28:51.439979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436295 5801 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:28:51.439979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436298 5801 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:28:51.439979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436300 5801 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:28:51.439979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436303 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:28:51.439979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436305 5801 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:28:51.439979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436308 5801 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:28:51.439979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436312 5801 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:28:51.439979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436797 5801 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:28:51.439979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436803 5801 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:28:51.439979 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436806 5801 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:28:51.440449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436809 5801 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:28:51.440449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436812 5801 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:28:51.440449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436815 5801 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:28:51.440449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436817 5801 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:28:51.440449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436820 5801 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:28:51.440449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436822 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:28:51.440449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436825 5801 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:28:51.440449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436828 5801 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:28:51.440449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436831 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:28:51.440449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436833 5801 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:28:51.440449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436836 5801 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:28:51.440449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436838 5801 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:28:51.440449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436841 5801 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:28:51.440449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436844 5801 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:28:51.440449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436846 5801 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:28:51.440449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436849 5801 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:28:51.440449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436851 5801 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:28:51.440449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436854 5801 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:28:51.440449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436857 5801 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:28:51.440449 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436859 5801 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:28:51.440949 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436861 5801 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:28:51.440949 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436865 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:28:51.440949 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436868 5801 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:28:51.440949 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436871 5801 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:28:51.440949 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436873 5801 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:28:51.440949 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436876 5801 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:28:51.440949 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436879 5801 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:28:51.440949 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436887 5801 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:28:51.440949 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436890 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:28:51.440949 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436893 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:28:51.440949 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436896 5801 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:28:51.440949 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436898 5801 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:28:51.440949 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436901 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:28:51.440949 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436903 5801 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:28:51.440949 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436906 5801 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:28:51.440949 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436909 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:28:51.440949 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436912 5801 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:28:51.440949 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436914 5801 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:28:51.440949 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436917 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:28:51.441414 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436920 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:28:51.441414 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436922 5801 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:28:51.441414 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436925 5801 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:28:51.441414 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436927 5801 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:28:51.441414 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436930 5801 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:28:51.441414 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436932 5801 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:28:51.441414 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436935 5801 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:28:51.441414 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436937 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:28:51.441414 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436940 5801 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:28:51.441414 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436942 5801 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:28:51.441414 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436945 5801 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:28:51.441414 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436948 5801 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:28:51.441414 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436950 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:28:51.441414 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436953 5801 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:28:51.441414 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436956 5801 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:28:51.441414 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436959 5801 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:28:51.441414 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436962 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:28:51.441414 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436964 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:28:51.441414 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436967 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:28:51.441414 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436970 5801 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:28:51.441908 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436975 5801 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:28:51.441908 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436977 5801 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:28:51.441908 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436980 5801 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:28:51.441908 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436982 5801 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:28:51.441908 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436985 5801 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:28:51.441908 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436987 5801 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:28:51.441908 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436990 5801 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:28:51.441908 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436993 5801 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:28:51.441908 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436996 5801 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:28:51.441908 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.436998 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:28:51.441908 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437001 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:28:51.441908 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437003 5801 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:28:51.441908 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437006 5801 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:28:51.441908 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437008 5801 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:28:51.441908 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437011 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:28:51.441908 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437014 5801 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:28:51.441908 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437016 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:28:51.441908 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437019 5801 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:28:51.441908 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437021 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:28:51.441908 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437024 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:28:51.442394 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437029 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:28:51.442394 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437031 5801 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:28:51.442394 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437034 5801 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:28:51.442394 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437036 5801 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:28:51.442394 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437039 5801 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:28:51.442394 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437042 5801 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:28:51.442394 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437044 5801 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:28:51.442394 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437047 5801 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:28:51.442394 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437050 5801 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:28:51.442394 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437054 5801 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:28:51.442394 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437058 5801 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:28:51.442394 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437062 5801 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:28:51.442394 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437068 5801 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:28:51.442394 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437071 5801 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:28:51.442394 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437073 5801 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:28:51.442394 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437077 5801 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:28:51.442394 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437080 5801 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:28:51.442394 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437082 5801 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:28:51.442394 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437085 5801 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:28:51.443193 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437087 5801 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:28:51.443193 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437090 5801 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:28:51.443193 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437092 5801 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:28:51.443193 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437095 5801 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:28:51.443193 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437097 5801 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:28:51.443193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437188 5801 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:28:51.443193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437196 5801 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:28:51.443193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437202 5801 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:28:51.443193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437207 5801 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:28:51.443193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437211 5801 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:28:51.443193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437214 5801 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:28:51.443193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437218 5801 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:28:51.443193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437223 5801 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:28:51.443193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437231 5801 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:28:51.443193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437234 5801 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:28:51.443193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437238 5801 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:28:51.443193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437241 5801 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:28:51.443193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437244 5801 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:28:51.443193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437247 5801 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:28:51.443193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437250 5801 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:28:51.443193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437253 5801 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:28:51.443193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437256 5801 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:28:51.443193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437259 5801 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:28:51.443193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437263 5801 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:28:51.443858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437266 5801 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:28:51.443858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437271 5801 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:28:51.443858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437274 5801 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:28:51.443858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437277 5801 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:28:51.443858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437282 5801 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:28:51.443858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437285 5801 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:28:51.443858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437288 5801 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:28:51.443858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437291 5801 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:28:51.443858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437294 5801 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:28:51.443858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437297 5801 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:28:51.443858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437301 5801 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:28:51.443858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437304 5801 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:28:51.443858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437307 5801 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:28:51.443858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437311 5801 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:28:51.443858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437314 5801 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:28:51.443858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437317 5801 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:28:51.443858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437320 5801 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:28:51.443858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437323 5801 flags.go:64] FLAG: --enable-server="true" Oct 30 21:28:51.443858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437326 5801 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:28:51.443858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437333 5801 flags.go:64] FLAG: --event-burst="100" Oct 30 21:28:51.443858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437337 5801 flags.go:64] FLAG: --event-qps="50" Oct 30 21:28:51.443858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437341 5801 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:28:51.443858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437344 5801 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:28:51.443858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437347 5801 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:28:51.444442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437352 5801 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:28:51.444442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437355 5801 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:28:51.444442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437358 5801 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:28:51.444442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437361 5801 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:28:51.444442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437364 5801 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:28:51.444442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437367 5801 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:28:51.444442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437370 5801 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:28:51.444442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437379 5801 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:28:51.444442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437382 5801 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:28:51.444442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437386 5801 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:28:51.444442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437389 5801 flags.go:64] FLAG: --feature-gates="" Oct 30 21:28:51.444442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437393 5801 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:28:51.444442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437397 5801 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:28:51.444442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437400 5801 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:28:51.444442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437403 5801 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:28:51.444442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437406 5801 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:28:51.444442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437409 5801 flags.go:64] FLAG: --help="false" Oct 30 21:28:51.444442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437412 5801 flags.go:64] FLAG: --hostname-override="ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:28:51.444442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437416 5801 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:28:51.444442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437419 5801 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:28:51.444442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437422 5801 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:28:51.444442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437425 5801 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:28:51.444442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437429 5801 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437432 5801 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437435 5801 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437437 5801 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437440 5801 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437443 5801 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437447 5801 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437451 5801 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437454 5801 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437457 5801 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437460 5801 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437463 5801 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437466 5801 flags.go:64] FLAG: --lock-file="" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437469 5801 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437472 5801 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437475 5801 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437480 5801 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437484 5801 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437497 5801 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437501 5801 flags.go:64] FLAG: --logging-format="text" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437504 5801 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437508 5801 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437511 5801 flags.go:64] FLAG: --manifest-url="" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437513 5801 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437518 5801 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:28:51.445120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437521 5801 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437525 5801 flags.go:64] FLAG: --max-pods="110" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437528 5801 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437531 5801 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437533 5801 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437537 5801 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437539 5801 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437542 5801 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437545 5801 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437553 5801 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437556 5801 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437559 5801 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437562 5801 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437565 5801 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437569 5801 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437572 5801 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437576 5801 flags.go:64] FLAG: --port="10250" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437579 5801 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437582 5801 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0ed923fb3a7a56b27" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437585 5801 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437588 5801 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437591 5801 flags.go:64] FLAG: --register-node="true" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437594 5801 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437598 5801 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437601 5801 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437604 5801 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:28:51.445727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437617 5801 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437621 5801 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437624 5801 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437627 5801 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437630 5801 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437633 5801 flags.go:64] FLAG: --runonce="false" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437636 5801 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437639 5801 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437642 5801 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437645 5801 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437648 5801 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437651 5801 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437654 5801 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437657 5801 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437660 5801 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437663 5801 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437666 5801 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437669 5801 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437672 5801 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437675 5801 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437679 5801 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437685 5801 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437688 5801 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437691 5801 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437698 5801 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:28:51.446393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437700 5801 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:28:51.447022 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437703 5801 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:28:51.447022 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437706 5801 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:28:51.447022 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437710 5801 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:28:51.447022 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437713 5801 flags.go:64] FLAG: --v="2" Oct 30 21:28:51.447022 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437717 5801 flags.go:64] FLAG: --version="false" Oct 30 21:28:51.447022 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437721 5801 flags.go:64] FLAG: --vmodule="" Oct 30 21:28:51.447022 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437727 5801 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:28:51.447022 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.437731 5801 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:28:51.447022 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437880 5801 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:28:51.447022 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437883 5801 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:28:51.447022 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437887 5801 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:28:51.447022 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437889 5801 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:28:51.447022 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437892 5801 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:28:51.447022 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437895 5801 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:28:51.447022 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437898 5801 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:28:51.447022 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437901 5801 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:28:51.447022 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437903 5801 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:28:51.447022 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437906 5801 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:28:51.447022 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437909 5801 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:28:51.447022 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437911 5801 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:28:51.447022 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437914 5801 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:28:51.447540 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437917 5801 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:28:51.447540 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437919 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:28:51.447540 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437922 5801 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:28:51.447540 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437925 5801 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:28:51.447540 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437927 5801 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:28:51.447540 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437931 5801 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:28:51.447540 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437934 5801 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:28:51.447540 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437937 5801 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:28:51.447540 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437939 5801 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:28:51.447540 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437942 5801 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:28:51.447540 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437944 5801 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:28:51.447540 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437947 5801 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:28:51.447540 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437950 5801 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:28:51.447540 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437952 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:28:51.447540 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437955 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:28:51.447540 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437957 5801 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:28:51.447540 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437960 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:28:51.447540 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437963 5801 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:28:51.447540 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437966 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:28:51.448134 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437969 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:28:51.448134 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437972 5801 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:28:51.448134 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437975 5801 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:28:51.448134 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437978 5801 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:28:51.448134 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437981 5801 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:28:51.448134 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437983 5801 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:28:51.448134 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437986 5801 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:28:51.448134 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437988 5801 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:28:51.448134 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437991 5801 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:28:51.448134 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437994 5801 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:28:51.448134 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437996 5801 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:28:51.448134 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.437999 5801 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:28:51.448134 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438001 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:28:51.448134 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438004 5801 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:28:51.448134 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438007 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:28:51.448134 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438010 5801 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:28:51.448134 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438013 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:28:51.448134 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438016 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:28:51.448134 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438019 5801 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:28:51.448134 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438022 5801 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:28:51.448662 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438025 5801 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:28:51.448662 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438027 5801 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:28:51.448662 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438030 5801 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:28:51.448662 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438033 5801 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:28:51.448662 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438035 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:28:51.448662 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438038 5801 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:28:51.448662 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438040 5801 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:28:51.448662 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438043 5801 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:28:51.448662 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438045 5801 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:28:51.448662 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438048 5801 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:28:51.448662 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438051 5801 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:28:51.448662 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438054 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:28:51.448662 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438057 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:28:51.448662 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438062 5801 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:28:51.448662 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438065 5801 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:28:51.448662 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438068 5801 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:28:51.448662 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438070 5801 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:28:51.448662 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438073 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:28:51.448662 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438075 5801 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:28:51.449181 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438078 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:28:51.449181 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438081 5801 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:28:51.449181 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438083 5801 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:28:51.449181 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438085 5801 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:28:51.449181 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438088 5801 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:28:51.449181 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438091 5801 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:28:51.449181 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438093 5801 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:28:51.449181 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438096 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:28:51.449181 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438099 5801 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:28:51.449181 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438103 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:28:51.449181 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438106 5801 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:28:51.449181 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438110 5801 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:28:51.449181 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438113 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:28:51.449181 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438115 5801 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:28:51.449181 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438118 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:28:51.449181 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438120 5801 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:28:51.449181 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438123 5801 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:28:51.449181 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438126 5801 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:28:51.449181 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438128 5801 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:28:51.449181 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438131 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:28:51.449672 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438133 5801 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:28:51.449672 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438136 5801 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:28:51.449672 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438138 5801 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:28:51.449672 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438141 5801 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:28:51.449672 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438144 5801 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:28:51.449672 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438147 5801 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:28:51.449672 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438149 5801 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:28:51.449672 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438152 5801 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:28:51.449672 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438155 5801 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:28:51.449672 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438158 5801 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:28:51.449672 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438160 5801 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:28:51.449672 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438163 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:28:51.449672 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438166 5801 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:28:51.449672 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438169 5801 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:28:51.449672 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438173 5801 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:28:51.449672 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.438197 5801 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:28:51.450186 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438237 5801 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:28:51.450186 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438241 5801 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:28:51.450186 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438244 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:28:51.450186 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438253 5801 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:28:51.450186 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438256 5801 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:28:51.450186 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438259 5801 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:28:51.450186 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438261 5801 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:28:51.450186 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438265 5801 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:28:51.450186 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438268 5801 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:28:51.450186 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438270 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:28:51.450186 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438273 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:28:51.450186 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438275 5801 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:28:51.450186 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438278 5801 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:28:51.450186 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438281 5801 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:28:51.450186 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438283 5801 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:28:51.450186 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438286 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:28:51.450186 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438288 5801 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:28:51.450186 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438291 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:28:51.450186 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438293 5801 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:28:51.450709 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438296 5801 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:28:51.450709 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438298 5801 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:28:51.450709 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438301 5801 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:28:51.450709 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438304 5801 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:28:51.450709 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438307 5801 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:28:51.450709 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438315 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:28:51.450709 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438319 5801 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:28:51.450709 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438324 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:28:51.450709 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438328 5801 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:28:51.450709 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438330 5801 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:28:51.450709 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438333 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:28:51.450709 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438335 5801 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:28:51.450709 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438338 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:28:51.450709 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438340 5801 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:28:51.450709 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438343 5801 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:28:51.450709 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438346 5801 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:28:51.450709 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438348 5801 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:28:51.450709 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438351 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:28:51.450709 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438353 5801 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:28:51.450709 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438356 5801 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:28:51.451247 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438360 5801 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:28:51.451247 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438362 5801 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:28:51.451247 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438365 5801 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:28:51.451247 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438368 5801 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:28:51.451247 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438370 5801 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:28:51.451247 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438373 5801 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:28:51.451247 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438375 5801 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:28:51.451247 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438378 5801 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:28:51.451247 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438380 5801 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:28:51.451247 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438383 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:28:51.451247 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438385 5801 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:28:51.451247 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438389 5801 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:28:51.451247 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438392 5801 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:28:51.451247 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438395 5801 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:28:51.451247 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438398 5801 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:28:51.451247 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438401 5801 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:28:51.451247 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438403 5801 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:28:51.451247 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438406 5801 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:28:51.451247 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438409 5801 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:28:51.451720 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438412 5801 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:28:51.451720 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438415 5801 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:28:51.451720 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438417 5801 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:28:51.451720 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438420 5801 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:28:51.451720 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438423 5801 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:28:51.451720 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438425 5801 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:28:51.451720 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438428 5801 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:28:51.451720 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438430 5801 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:28:51.451720 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438433 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:28:51.451720 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438435 5801 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:28:51.451720 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438438 5801 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:28:51.451720 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438440 5801 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:28:51.451720 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438443 5801 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:28:51.451720 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438446 5801 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:28:51.451720 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438449 5801 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:28:51.451720 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438451 5801 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:28:51.451720 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438454 5801 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:28:51.451720 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438457 5801 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:28:51.451720 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438461 5801 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:28:51.451720 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438463 5801 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:28:51.452305 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438466 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:28:51.452305 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438469 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:28:51.452305 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438478 5801 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:28:51.452305 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438481 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:28:51.452305 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438484 5801 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:28:51.452305 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438487 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:28:51.452305 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438490 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:28:51.452305 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438493 5801 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:28:51.452305 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438496 5801 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:28:51.452305 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438499 5801 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:28:51.452305 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438502 5801 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:28:51.452305 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438504 5801 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:28:51.452305 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438507 5801 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:28:51.452305 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438510 5801 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:28:51.452305 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438513 5801 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:28:51.452305 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438515 5801 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:28:51.452305 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438518 5801 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:28:51.452305 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438520 5801 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:28:51.452305 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438523 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:28:51.452305 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438527 5801 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:28:51.452806 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438529 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:28:51.452806 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438532 5801 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:28:51.452806 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438535 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:28:51.452806 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438537 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:28:51.452806 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438540 5801 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:28:51.452806 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438544 5801 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:28:51.452806 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438547 5801 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:28:51.452806 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.438549 5801 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:28:51.452806 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.442149 5801 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:28:51.452806 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.442165 5801 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:28:51.452806 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.442184 5801 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:28:51.452806 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.442189 5801 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:28:51.452806 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442242 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:28:51.452806 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442247 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:28:51.452806 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442250 5801 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:28:51.452806 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442253 5801 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:28:51.452806 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442256 5801 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:28:51.452806 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442259 5801 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:28:51.452806 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442261 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:28:51.453273 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442264 5801 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:28:51.453273 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442267 5801 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:28:51.453273 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442269 5801 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:28:51.453273 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442272 5801 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:28:51.453273 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442275 5801 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:28:51.453273 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442277 5801 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:28:51.453273 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442280 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:28:51.453273 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442283 5801 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:28:51.453273 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442286 5801 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:28:51.453273 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442288 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:28:51.453273 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442291 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:28:51.453273 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442293 5801 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:28:51.453273 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442296 5801 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:28:51.453273 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442298 5801 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:28:51.453273 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442301 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:28:51.453273 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442304 5801 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:28:51.453273 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442306 5801 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:28:51.453273 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442309 5801 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:28:51.453273 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442312 5801 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:28:51.453726 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442315 5801 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:28:51.453726 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442317 5801 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:28:51.453726 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442320 5801 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:28:51.453726 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442355 5801 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:28:51.453726 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442400 5801 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:28:51.453726 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442489 5801 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:28:51.453726 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442496 5801 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:28:51.453726 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442502 5801 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:28:51.453726 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442507 5801 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:28:51.453726 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442512 5801 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:28:51.453726 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442517 5801 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:28:51.453726 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442522 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:28:51.453726 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442527 5801 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:28:51.453726 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442531 5801 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:28:51.453726 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442536 5801 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:28:51.453726 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442541 5801 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:28:51.453726 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442545 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:28:51.453726 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442555 5801 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:28:51.453726 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442560 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:28:51.453726 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442564 5801 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:28:51.454281 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442569 5801 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:28:51.454281 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442573 5801 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:28:51.454281 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442577 5801 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:28:51.454281 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442582 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:28:51.454281 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442586 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:28:51.454281 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442591 5801 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:28:51.454281 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442596 5801 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:28:51.454281 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442600 5801 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:28:51.454281 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442604 5801 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:28:51.454281 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442611 5801 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:28:51.454281 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442622 5801 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:28:51.454281 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442627 5801 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:28:51.454281 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442631 5801 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:28:51.454281 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442636 5801 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:28:51.454281 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442640 5801 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:28:51.454281 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442644 5801 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:28:51.454281 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442649 5801 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:28:51.454281 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442653 5801 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:28:51.454281 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442659 5801 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:28:51.454281 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442664 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:28:51.454776 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442668 5801 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:28:51.454776 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442673 5801 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:28:51.454776 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442682 5801 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:28:51.454776 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442687 5801 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:28:51.454776 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442692 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:28:51.454776 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442696 5801 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:28:51.454776 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442701 5801 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:28:51.454776 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442706 5801 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:28:51.454776 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442710 5801 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:28:51.454776 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442715 5801 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:28:51.454776 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442719 5801 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:28:51.454776 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442723 5801 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:28:51.454776 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442728 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:28:51.454776 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442732 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:28:51.454776 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442737 5801 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:28:51.454776 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442745 5801 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:28:51.454776 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442765 5801 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:28:51.454776 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442770 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:28:51.454776 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442775 5801 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:28:51.454776 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442780 5801 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:28:51.455261 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442785 5801 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:28:51.455261 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442789 5801 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:28:51.455261 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442794 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:28:51.455261 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442798 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:28:51.455261 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442803 5801 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:28:51.455261 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442809 5801 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:28:51.455261 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442817 5801 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:28:51.455261 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442827 5801 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:28:51.455261 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442836 5801 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:28:51.455261 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442841 5801 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:28:51.455261 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442845 5801 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:28:51.455261 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442850 5801 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:28:51.455261 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442854 5801 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:28:51.455261 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442859 5801 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:28:51.455261 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442863 5801 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:28:51.455261 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442868 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:28:51.455261 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442872 5801 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:28:51.455261 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442876 5801 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:28:51.455261 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442881 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:28:51.455710 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.442885 5801 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:28:51.455710 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.442943 5801 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:28:51.455710 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443459 5801 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:28:51.455710 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443583 5801 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:28:51.455710 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443593 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:28:51.455710 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443599 5801 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:28:51.455710 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443603 5801 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:28:51.455710 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443611 5801 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:28:51.455710 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443618 5801 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:28:51.455710 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443625 5801 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:28:51.455710 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443629 5801 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:28:51.455710 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443634 5801 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:28:51.455710 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443639 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:28:51.455710 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443643 5801 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:28:51.455710 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443647 5801 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:28:51.455710 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443652 5801 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:28:51.456120 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443656 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:28:51.456120 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443660 5801 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:28:51.456120 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443665 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:28:51.456120 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443669 5801 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:28:51.456120 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443673 5801 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:28:51.456120 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443678 5801 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:28:51.456120 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443684 5801 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:28:51.456120 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443690 5801 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:28:51.456120 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443694 5801 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:28:51.456120 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443700 5801 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:28:51.456120 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443704 5801 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:28:51.456120 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443708 5801 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:28:51.456120 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443713 5801 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:28:51.456120 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443718 5801 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:28:51.456120 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443722 5801 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:28:51.456120 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443726 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:28:51.456120 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443731 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:28:51.456120 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443735 5801 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:28:51.456120 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443740 5801 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:28:51.456628 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443745 5801 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:28:51.456628 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443763 5801 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:28:51.456628 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443768 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:28:51.456628 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443772 5801 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:28:51.456628 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443781 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:28:51.456628 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443785 5801 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:28:51.456628 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443791 5801 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:28:51.456628 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443795 5801 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:28:51.456628 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443799 5801 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:28:51.456628 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443805 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:28:51.456628 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443809 5801 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:28:51.456628 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443813 5801 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:28:51.456628 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443818 5801 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:28:51.456628 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443823 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:28:51.456628 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443830 5801 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:28:51.456628 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443835 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:28:51.456628 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443842 5801 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:28:51.456628 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443847 5801 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:28:51.456628 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443852 5801 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:28:51.457160 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443856 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:28:51.457160 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443859 5801 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:28:51.457160 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443862 5801 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:28:51.457160 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443866 5801 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:28:51.457160 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443869 5801 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:28:51.457160 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443871 5801 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:28:51.457160 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443874 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:28:51.457160 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443877 5801 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:28:51.457160 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443880 5801 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:28:51.457160 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443883 5801 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:28:51.457160 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443885 5801 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:28:51.457160 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443888 5801 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:28:51.457160 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443891 5801 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:28:51.457160 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443894 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:28:51.457160 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443897 5801 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:28:51.457160 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443900 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:28:51.457160 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443903 5801 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:28:51.457160 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443907 5801 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:28:51.457160 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443910 5801 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:28:51.457160 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443913 5801 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:28:51.457658 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443916 5801 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:28:51.457658 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443918 5801 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:28:51.457658 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443921 5801 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:28:51.457658 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443923 5801 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:28:51.457658 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443926 5801 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:28:51.457658 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443929 5801 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:28:51.457658 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443932 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:28:51.457658 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443934 5801 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:28:51.457658 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443937 5801 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:28:51.457658 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443940 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:28:51.457658 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443944 5801 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:28:51.457658 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443947 5801 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:28:51.457658 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443950 5801 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:28:51.457658 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443953 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:28:51.457658 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443956 5801 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:28:51.457658 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443959 5801 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:28:51.457658 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443962 5801 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:28:51.457658 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443965 5801 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:28:51.457658 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443967 5801 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:28:51.458140 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443970 5801 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:28:51.458140 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443972 5801 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:28:51.458140 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443975 5801 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:28:51.458140 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443978 5801 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:28:51.458140 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443980 5801 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:28:51.458140 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443983 5801 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:28:51.458140 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443985 5801 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:28:51.458140 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443988 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:28:51.458140 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443991 5801 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:28:51.458140 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443993 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:28:51.458140 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.443997 5801 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:28:51.458140 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444000 5801 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:28:51.458140 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444003 5801 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:28:51.458140 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444006 5801 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:28:51.458140 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444008 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:28:51.458140 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.444044 5801 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:28:51.458541 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444082 5801 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:28:51.458541 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444086 5801 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:28:51.458541 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444090 5801 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:28:51.458541 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444093 5801 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:28:51.458541 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444095 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:28:51.458541 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444098 5801 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:28:51.458541 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444100 5801 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:28:51.458541 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444105 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:28:51.458541 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444107 5801 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:28:51.458541 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444110 5801 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:28:51.458541 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444114 5801 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:28:51.458541 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444118 5801 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:28:51.458541 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444121 5801 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:28:51.458541 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444124 5801 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:28:51.458541 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444126 5801 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:28:51.458541 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444129 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:28:51.458541 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444131 5801 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:28:51.458541 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444134 5801 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:28:51.458541 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444137 5801 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:28:51.458541 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444139 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:28:51.459071 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444142 5801 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:28:51.459071 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444145 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:28:51.459071 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444147 5801 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:28:51.459071 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444150 5801 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:28:51.459071 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444152 5801 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:28:51.459071 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444156 5801 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:28:51.459071 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444161 5801 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:28:51.459071 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444165 5801 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:28:51.459071 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444169 5801 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:28:51.459071 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444172 5801 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:28:51.459071 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444175 5801 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:28:51.459071 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444178 5801 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:28:51.459071 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444181 5801 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:28:51.459071 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444184 5801 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:28:51.459071 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444186 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:28:51.459071 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444189 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:28:51.459071 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444192 5801 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:28:51.459071 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444194 5801 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:28:51.459071 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444197 5801 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:28:51.459536 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444201 5801 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:28:51.459536 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444204 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:28:51.459536 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444207 5801 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:28:51.459536 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444209 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:28:51.459536 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444212 5801 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:28:51.459536 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444215 5801 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:28:51.459536 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444218 5801 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:28:51.459536 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444220 5801 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:28:51.459536 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444223 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:28:51.459536 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444225 5801 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:28:51.459536 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444228 5801 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:28:51.459536 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444231 5801 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:28:51.459536 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444233 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:28:51.459536 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444236 5801 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:28:51.459536 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444238 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:28:51.459536 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444241 5801 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:28:51.459536 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444243 5801 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:28:51.459536 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444246 5801 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:28:51.459536 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444249 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:28:51.460017 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444251 5801 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:28:51.460017 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444255 5801 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:28:51.460017 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444258 5801 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:28:51.460017 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444260 5801 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:28:51.460017 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444263 5801 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:28:51.460017 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444266 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:28:51.460017 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444269 5801 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:28:51.460017 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444272 5801 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:28:51.460017 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444275 5801 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:28:51.460017 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444278 5801 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:28:51.460017 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444280 5801 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:28:51.460017 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444283 5801 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:28:51.460017 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444285 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:28:51.460017 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444289 5801 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:28:51.460017 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444292 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:28:51.460017 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444296 5801 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:28:51.460017 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444298 5801 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:28:51.460017 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444301 5801 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:28:51.460017 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444304 5801 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:28:51.460017 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444306 5801 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:28:51.460534 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444309 5801 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:28:51.460534 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444311 5801 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:28:51.460534 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444314 5801 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:28:51.460534 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444317 5801 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:28:51.460534 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444320 5801 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:28:51.460534 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444322 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:28:51.460534 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444325 5801 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:28:51.460534 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444328 5801 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:28:51.460534 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444330 5801 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:28:51.460534 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444333 5801 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:28:51.460534 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444336 5801 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:28:51.460534 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444338 5801 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:28:51.460534 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444342 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:28:51.460534 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444346 5801 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:28:51.460534 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444349 5801 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:28:51.460534 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444351 5801 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:28:51.460534 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444354 5801 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:28:51.460534 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444356 5801 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:28:51.460534 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444359 5801 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:28:51.461003 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444362 5801 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:28:51.461003 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444365 5801 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:28:51.461003 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444367 5801 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:28:51.461003 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444370 5801 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:28:51.461003 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444373 5801 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:28:51.461003 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444375 5801 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:28:51.461003 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444379 5801 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:28:51.461003 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444382 5801 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:28:51.461003 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:28:51.444385 5801 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:28:51.461003 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.444443 5801 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:28:51.461003 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.445406 5801 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 21:28:51.461003 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.445473 5801 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 21:28:51.461003 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.445680 5801 server.go:1014] "Starting client certificate rotation" Oct 30 21:28:51.461003 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.445821 5801 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:28:51.461003 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.445870 5801 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:22:39 +0000 UTC" deadline="2028-05-30 11:10:04.357935525 +0000 UTC" Oct 30 21:28:51.461003 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.445904 5801 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13861h41m12.912033068s" Oct 30 21:28:51.461401 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.446199 5801 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:28:51.461401 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.447256 5801 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:28:51.461401 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.448376 5801 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:28:51.461401 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.450147 5801 log.go:25] "Validated CRI v1 image API" Oct 30 21:28:51.461401 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.450895 5801 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:28:51.461401 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.452866 5801 fs.go:135] Filesystem UUIDs: map[192550a1-7e00-4901-b523-e4a83ba1c9e8:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a24e2d3f-c530-4c95-913f-b50036f8c3dc:/dev/nvme0n1p4] Oct 30 21:28:51.461601 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.452887 5801 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01498921daece598a7ce2734824ebc2fd4b2b973c4e826a4e09f9a58838d32b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01498921daece598a7ce2734824ebc2fd4b2b973c4e826a4e09f9a58838d32b9/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/04996af222f21f33acf2dbb3b6b94b8be5ca45bb4627d0208b7720a427210892/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/04996af222f21f33acf2dbb3b6b94b8be5ca45bb4627d0208b7720a427210892/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f3721f94fe154516ba62cca3e2cbe1788b0776dd7701f70a95539931b316fef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f3721f94fe154516ba62cca3e2cbe1788b0776dd7701f70a95539931b316fef/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/253215521b57eb57279a0ac998f8cbcd172f5d06f7d99cad8392adf60220daeb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/253215521b57eb57279a0ac998f8cbcd172f5d06f7d99cad8392adf60220daeb/userdata/shm major:0 minor:319 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2fe33a545cb5bbcb29c98119be301ae623972d70eff09fbb658bc8dd3ecd78c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2fe33a545cb5bbcb29c98119be301ae623972d70eff09fbb658bc8dd3ecd78c5/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34905c4138e89ebabc77c38a62d4d277a3e9ad38e44c156e43ee57012e0deae7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34905c4138e89ebabc77c38a62d4d277a3e9ad38e44c156e43ee57012e0deae7/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4067ec29eb4ea5cbff18adbbd3d2f6346dfa9ffdc48fbb0744ac13476c2fb181/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4067ec29eb4ea5cbff18adbbd3d2f6346dfa9ffdc48fbb0744ac13476c2fb181/userdata/shm major:0 minor:303 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4219a488b10a8b86b8a9c029c3d199ff40e2ffda7684bf1ea381382f34a1023e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4219a488b10a8b86b8a9c029c3d199ff40e2ffda7684bf1ea381382f34a1023e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46ea9f9182e771ae4b2a53ab78cc99399a8f3eba9f63301332e3064c96a24c69/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46ea9f9182e771ae4b2a53ab78cc99399a8f3eba9f63301332e3064c96a24c69/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/54fd1c126cabbff9f892111cc0bbff2664cb51ea952375ab1eb54b8401f6cbf8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/54fd1c126cabbff9f892111cc0bbff2664cb51ea952375ab1eb54b8401f6cbf8/userdata/shm major:0 minor:383 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/593202543ae57c5e203fa3abb539eb711165e48b2bf2ba5cef1721e0bcc17287/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/593202543ae57c5e203fa3abb539eb711165e48b2bf2ba5cef1721e0bcc17287/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/69334d317041ccb071934aed82b2a2668b57f54889017d16c8443ff3bfdaca9e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/69334d317041ccb071934aed82b2a2668b57f54889017d16c8443ff3bfdaca9e/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c95fd8aec2140a47abfc68a4ebcb6b3b50eec9e9579bc892c31a5fff26eb8e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c95fd8aec2140a47abfc68a4ebcb6b3b50eec9e9579bc892c31a5fff26eb8e7/userdata/shm major:0 minor:387 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/828aaed34bc5f0464c0e592518791a70af36bb4a367070d703daf52da7d54d26/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/828aaed34bc5f0464c0e592518791a70af36bb4a367070d703daf52da7d54d26/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a23d62cf087a22dbe4e7bf329a60a92adba8cb51a46683861e9baba4d8d3c18b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a23d62cf087a22dbe4e7bf329a60a92adba8cb51a46683861e9baba4d8d3c18b/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5510e59bc4d70b7949d7a6c736e5765e4da41eecb6bfbfed83b31498a7c5426/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5510e59bc4d70b7949d7a6c736e5765e4da41eecb6bfbfed83b31498a7c5426/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aafb99e79b91dc43038e1c3d53e6955437495733c43e838913a87c6b2f17a553/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aafb99e79b91dc43038e1c3d53e6955437495733c43e838913a87c6b2f17a553/userdata/shm major:0 minor:390 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/acb30153b006b8ad51cb2725eeeb41d1406ddabeab22c41da892be6125a8d85a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/acb30153b006b8ad51cb2725eeeb41d1406ddabeab22c41da892be6125a8d85a/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bcd8b205dfbf9b7c1e054114904f54e7451342bcfa11344e056753f200b7694b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bcd8b205dfbf9b7c1e054114904f54e7451342bcfa11344e056753f200b7694b/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d7b9f6a71d269f8ddf38e89179e4d4b9b8da09b774010f87c7d5614ddb1043f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d7b9f6a71d269f8ddf38e89179e4d4b9b8da09b774010f87c7d5614ddb1043f7/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dfad36cb6e1ed10a6cd6660f40a16c252c832f5e86b10f65345c745eaaae034d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dfad36cb6e1ed10a6cd6660f40a16c252c832f5e86b10f65345c745eaaae034d/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/edbef9c655815b9877f936c73f1f75f3449440552d90723d5d0a8f2b9727a8da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/edbef9c655815b9877f936c73f1f75f3449440552d90723d5d0a8f2b9727a8da/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/183318eb-952e-4e93-b716-6ab50b9c58dd/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/183318eb-952e-4e93-b716-6ab50b9c58dd/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/189b46e5-b36a-4ce4-be3e-b5486b7e45a8/volumes/kubernetes.io~projected/kube-api-access-sl65r:{mountpoint:/var/lib/kubelet/pods/189b46e5-b36a-4ce4-be3e-b5486b7e45a8/volumes/kubernetes.io~projected/kube-api-access-sl65r major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/189b46e5-b36a-4ce4-be3e-b5486b7e45a8/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/189b46e5-b36a-4ce4-be3e-b5486b7e45a8/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/189b46e5-b36a-4ce4-be3e-b5486b7e45a8/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/189b46e5-b36a-4ce4-be3e-b5486b7e45a8/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21593b07-35ab-409e-98ef-74167147e27c/volumes/kubernetes.io~projected/kube-api-access-lz24x:{mountpoint:/var/lib/kubelet/pods/21593b07-35ab-409e-98ef-74167147e27c/volumes/kubernetes.io~projected/kube-api-access-lz24x major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21593b07-35ab-409e-98ef-74167147e27c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/21593b07-35ab-409e-98ef-74167147e27c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21593b07-35ab-409e-98ef-74167147e27c/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/21593b07-35ab-409e-98ef-74167147e27c/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ae36e61-388d-4bc0-8810-aae71f3703c6/volumes/kubernetes.io~projected/kube-api-access-vwp8n:{mountpoint:/var/lib/kubelet/pods/2ae36e61-388d-4bc0-8810-aae71f3703c6/volumes/kubernetes.io~projected/kube-api-access-vwp8n major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78/volumes/kubernetes.io~projected/kube-api-access-4gstg:{mountpoint:/var/lib/kubelet/pods/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78/volumes/kubernetes.io~projected/kube-api-access-4gstg major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78/volumes/kubernetes.io~secret/metrics-certs major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5934c36a-2955-4f0c-a8e8-615f2fa07454/volumes/kubernetes.io~projected/kube-api-access-mx5l7:{mountpoint:/var/lib/kubelet/pods/5934c36a-2955-4f0c-a8e8-615f2fa07454/volumes/kubernetes.io~projected/kube-api-access-mx5l7 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8/volumes/kubernetes.io~projected/kube-api-access-krfth:{mountpoint:/var/lib/kubelet/pods/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8/volumes/kubernetes.io~projected/kube-api-access-krfth major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c702aef-eccc-4da1-8040-c860c607a466/volumes/kubernetes.io~projected/kube-api-access-p2vfb:{mountpoint:/var/lib/kubelet/pods/5c702aef-eccc-4da1-8040-c860c607a466/volumes/kubernetes.io~projected/kube-api-access-p2vfb major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c702aef-eccc-4da1-8040-c860c607a466/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/5c702aef-eccc-4da1-8040-c860c607a466/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6841b7e6-42ec-493f-9613-ac4902212fac/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/6841b7e6-42ec-493f-9613-ac4902212fac/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6841b7e6-42ec-493f-9613-ac4902212fac/volumes/kubernetes.io~projected/kube-api-access-2dpq2:{mountpoint:/var/lib/kubelet/pods/6841b7e6-42ec-493f-9613-ac4902212fac/volumes/kubernetes.io~projected/kube-api-access-2dpq2 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6841b7e6-42ec-493f-9613-ac4902212fac/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/6841b7e6-42ec-493f-9613-ac4902212fac/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f1829b1-8087-46b1-b64d-e3f15811caf6/volumes/kubernetes.io~projected/kube-api-access-z27h9:{mountpoint:/var/lib/kubelet/pods/7f1829b1-8087-46b1-b64d-e3f15811caf6/volumes/kubernetes.io~projected/kube-api-access-z27h9 major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f1829b1-8087-46b1-b64d-e3f15811caf6/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/7f1829b1-8087-46b1-b64d-e3f15811caf6/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f1829b1-8087-46b1-b64d-e3f15811caf6/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/7f1829b1-8087-46b1-b64d-e3f15811caf6/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/808521e6-a0b3-4a24-b0b0-5a3cabffe0f5/volumes/kubernetes.io~projected/kube-api-access-7ztrc:{mountpoint:/var/lib/kubelet/pods/808521e6-a0b3-4a24-b0b0-5a3cabffe0f5/volumes/kubernetes.io~projected/kube-api-access-7ztrc major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/808521e6-a0b3-4a24-b0b0-5a3cabffe0f5/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/808521e6-a0b3-4a24-b0b0-5a3cabffe0f5/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a/volumes/kubernetes.io~projected/kube-api-access-d92bb:{mountpoint:/var/lib/kubelet/pods/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a/volumes/kubernetes.io~projected/kube-api-access-d92bb major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d43f954-119d-40c5-9727-1e25043da4ee/volumes/kubernetes.io~projected/kube-api-access-bczbc:{mountpoint:/var/lib/kubelet/pods/8d43f954-119d-40c5-9727-1e25043da4ee/volumes/kubernetes.io~projected/kube-api-access-bczbc major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d43f954-119d-40c5-9727-1e25043da4ee/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/8d43f954-119d-40c5-9727-1e25043da4ee/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d43f954-119d-40c5-9727-1e25043da4ee/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/8d43f954-119d-40c5-9727-1e25043da4ee/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/950f0225-e1c9-4c9c-8fc6-25dba5db7120/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/950f0225-e1c9-4c9c-8fc6-25dba5db7120/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volume-subpaths/web-config/config-reloader/2 major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volume-subpaths/web-config/prometheus/15 major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~empty-dir/config-out major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~projected/kube-api-access-4ppdt:{mountpoint:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~projected/kube-api-access-4ppdt major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~projected/tls-assets major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/config major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/web-config major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a31aa40d-e199-4095-b9b9-f2c7e31822e2/volumes/kubernetes.io~projected/kube-api-access-kqfx8:{mountpoint:/var/lib/kubelet/pods/a31aa40d-e199-4095-b9b9-f2c7e31822e2/volumes/kubernetes.io~projected/kube-api-access-kqfx8 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a31aa40d-e199-4095-b9b9-f2c7e31822e2/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/a31aa40d-e199-4095-b9b9-f2c7e31822e2/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb88c1ff-4b28-4812-9b2b-8876983720e5/volumes/kubernetes.io~projected/kube-api-access-qqcwl:{mountpoint:/var/lib/kubelet/pods/bb88c1ff-4b28-4812-9b2b-8876983720e5/volumes/kubernetes.io~projected/kube-api-access-qqcwl major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3af705e-b3ba-4379-8214-07ba1b431824/volumes/kubernetes.io~projected/kube-api-access-8sm8p:{mountpoint:/var/lib/kubelet/pods/c3af705e-b3ba-4379-8214-07ba1b431824/volumes/kubernetes.io~projected/kube-api-access-8sm8p major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4/volumes/kubernetes.io~projected/kube-api-access-2hgqm:{mountpoint:/var/lib/kubelet/pods/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4/volumes/kubernetes.io~projected/kube-api-access-2hgqm major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da5c17e9-1229-4d33-ba12-3eab4727e64d/volumes/kubernetes.io~projected/kube-api-access-rmtxv:{mountpoint:/var/lib/kubelet/pods/da5c17e9-1229-4d33-ba12-3eab4727e64d/volumes/kubernetes.io~projected/kube-api-access-rmtxv major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da5c17e9-1229-4d33-ba12-3eab4727e64d/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/da5c17e9-1229-4d33-ba12-3eab4727e64d/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da5c17e9-1229-4d33-ba12-3eab4727e64d/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/da5c17e9-1229-4d33-ba12-3eab4727e64d/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da5c17e9-1229-4d33-ba12-3eab4727e64d/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/da5c17e9-1229-4d33-ba12-3eab4727e64d/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da5c17e9-1229-4d33-ba12-3eab4727e64d/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/da5c17e9-1229-4d33-ba12-3eab4727e64d/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d/volumes/kubernetes.io~projected/kube-api-access-qjtk6:{mountpoint:/var/lib/kubelet/pods/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d/volumes/kubernetes.io~projected/kube-api-access-qjtk6 major:0 minor:67 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/cec55c407f794cd806bfb40ccc284c6d6767c69b63c117089bfe3884713f9398/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/6f0b85f58ea69d41723ca5e618bfc0dd1a2ca156a2b404e15d48bafa043bd9ce/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5df03b3881e430b09d142f29b21493c3542496f680db8cead6e3e2d54d32c13e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/ed7623150ad32a7da548bc966e3eded4e704ed4536acc6d7ae6d61bef672fad4/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/4606227585c66a7c4c5204d2d84425bb9832dc8bc3db23515fd934211031dd1f/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/568db5ea3e29595c463542f12f25123006cd16e652fb71ee819aee4c5bf8f1a9/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/e1b0be16ac162265f143ff9429e05a778d1a84250c0f3c0a50c64c5c77a65bf2/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/ac01b9d8db1cdd1e98e9052d4c67bb20d2a62320834754a4b1d5e681ae3268ae/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/4c4924bb1c8969f455287479370430bd38d56b684bd44426ba13377784222e67/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/b0ae57e89562438a3c2db6008d49c8280f666c1950e22b27d2481b6824667fe2/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/f1084ed5017ac805e0cde8878d46c2b6ab5eea9bb5cc89e775b6c04b0e5cf4d1/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/e38a81781323bf2575fdbd2e826ff82f57d315884f6bd53f309c252ca3176a41/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/da71ae9f94caa0c22beab294975208eda12c62701837248c82cdc07b2c9e02e0/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/36254e53618292c006c982e48278ead9e7323437d9c03c1382999fccd2574a85/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/6da84f6ffd38a1bb78c72eaf692db1b88e430463522f263380c2545c816867f9/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/15816adbe456a6dd7a07296371c285b9d4f03b3e315378efb8dac56d0effe15d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/eb0502f9048c0909a47e294470cbc8956f3cf340097dc16523aa4e994c913619/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/3e8c5a97067f16924392b2a9ae66f6bd3cc6f9c29e1ce35ff2bcbb808cf2ac81/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/aec5ebd90ed07d5a40a1f542dcce38957ca7c685fd5681911859a3ac149b41b8/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/96742d20810d33e547d1b4969216c93ad2989d5ed54f0056a610eeae40c1cd1a/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f429b74e06ca5f7f3e965ede78a7abef26d5d34b1caa223b9262d0653712cb85/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/2b2bc6c10784f1b3a762c267ac73869ed4659b1efc483a51bd4aab1473df09bc/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/daa11391b35f5ce6a059c65c4bf076bd25b0fd26b84266a1a6244b2b953b04a5/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/be2ca01acbbfcca5eecdcd42513f9a550c0277cf6f8933a0be97cb2426fa5d71/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/4cab22378ecbb119c8be894325c95c18009ecd752192c971bbc74fa897a70ca6/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/fc04c48fa856e3360fcb8156cc96f86ff84721d8e0772869f8871eb0646e0f53/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/5d89dbaf19c2df9b3aa0f217c5ff8cc681d749b08b0a0bb2720724088da95fdb/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/99f427763de9bbf14e10be783249ca37033416cd3e332a108acf1861eccf4ede/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/8dfa2cf9dc13ae3bfbc92094f643f7e8914668e3f1cb0cf73df70e817faf5617/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/de5750058763f7c8b501ce0f8721864b0e3d26eb27e46355bbfe96719eb38d71/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/12eae159de343585884650835bdaff0b4642818c64b362a0645c08fc60a6d6da/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/c9f50e449c4f8b04afcddcfdc633485de569cc798fb83f5cd0ec53fb3e4ad8b1/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/dceee8f1b9a3bd3b8b5edb8549b9a048290c1c23327fcfaee0485ca9468e5da3/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/de34051063e99b7395d2d26817f6bdc327ce952893026e176a8cc5cd55b432d0/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/64c517558ab31c2938bc3bcff7917dbaef03b65deba3c8f3a4095cf306f640cd/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/bc7ae283e820937bb5d9f1a8f65be7b6dc53639c83767d3169b7f99d275fca18/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/15870fe28fc1b51e2e0c1edd80892068a415600b3ac0292f404ce18b4c3101c8/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/867151a23715a7bcab35b574f3ccfa992690847824f39cfda185a3f61bcfab20/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/0fbbcadef3b07979f65cf24b9b62ef57f109d039e5a437de80c88c4202e325e3/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/4b2c70468da4f00fb082134ed4ebae2d0dc844025b05e90edce8930b93fe62ab/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/9b4cd29e5aa18b5d6a8904aa8ad312eb90d1b2b0bdb1adbed5eda4ce4f1c3112/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/f8d61ef629cb3de2edef97987745e248c2b1c80ec2a020ae81edd3d9cc18232b/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/4c34c6e049df6d6890630f1a85d5408ff921f8dc1aa24929c6042aa8b8dfe5fe/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/041bdea346c61405ba01342bc0c04aedd2dde08c7ae3acdfec8d577f3ac67d49/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/c029298a847880f0a4ba689eed25608ebf61f6af9c3f42576807f4046c209953/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/3eaf3b307c2f42a4252b4edeb1b0cbe45ef023d8035f21f7d1566a48c2cc25b0/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/2e93ecb7abc59bdcd525c345de6493b1ff1e51464ee3e17fcd333b16cd547675/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/436c524df4fe7c8c9a357da2df04d36d526bc4936fb3c179cd716b5c43a60b71/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/9d09ae54a7cf382c76464ee1cb846346a5b8d0c497052ae9b5b614de2aa28138/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-371:{mountpoint:/var/lib/containers/storage/overlay/4ca4013e38af83dcd02558f59ea7be28b86030802f3acebed807accb2c574fd5/merged major:0 minor:371 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/cd74fea285de84cc4e319e2d636fb47ea036d3d93031f05814c7ee9fb46d27b6/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/26f3c92b89d7e853de17dde432e9ae909f1d162c7446956c23c4df2b64e8c72c/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/277f104de711133063d6d7e97d413e3a6e0e3153edc44cd8dc54fc257d72fa54/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/47fc51872be30175d6149657fd7cbdab28adb005cb04851bab6c8dd97deb6d4a/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/8b9515a0c397f01113d72182c52bef87be6e325915f37c6406aa7bd201c5f4f5/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/fa1a6e57318f3de731c76e2b1ef1ac7a93cf522bdaf8578ff4d4ed4c95585ae6/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/0852b86acbd3e7a36da0266310ca90d3aaba10c6aeec5bda3fa5e6a851624892/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/13d74a7e841af2bb53403190e389cd9bb9f9ccc5039d934bd102e8f9f6d9455d/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/b7a9f178076b7c522e872be707a1d63e636d839a932966a1ac1503f18474d437/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/345b1a7568331e1a0b45977d821bbc317da5a04afe6c880e8464ceaac49775d1/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/01b7fa9d1fdf28b9bc1156c61fb755044750bdea7f73a4ff42a76996b2790019/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/caed00cd41744a3500e8f19ad80d07de29d3fbceefc5a45f93db624c36b81ed7/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/34cbacb906073e4f2713bb93a112b9852dd0f99a24f1dc490c329b120bb1199b/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/a1652c7a04d836e892f23b99996f7da1c389e12cbe5e8cfc9a51ba782b08b864/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/992f9e657eb00ded3414ca26d9e81342313bb8f2a93fa271c369712f1783a85e/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/04768880fe995150528d718f6b7da22be0c8758c863ce1afa851800601cf133b/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/000b3696d5920aaf678e412d13b1cef721be168580999fd0cdf2d123aa1027f1/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/c97249072a5531a0b76204a7b164ff3af07058ebd6aff78912e426db83215a31/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 21:28:51.461997 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.460919 5801 manager.go:217] Machine: {Timestamp:2026-10-30 21:28:51.460442134 +0000 UTC m=+0.056258787 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3122891 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2238eae22eb5ba9498d51db988d5b7 SystemUUID:ec2238ea-e22e-b5ba-9498-d51db988d5b7 BootID:20879828-f4b6-4d33-a897-273cc6bda5ac Filesystems:[{Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/69334d317041ccb071934aed82b2a2668b57f54889017d16c8443ff3bfdaca9e/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4219a488b10a8b86b8a9c029c3d199ff40e2ffda7684bf1ea381382f34a1023e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4/volumes/kubernetes.io~projected/kube-api-access-2hgqm DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7f1829b1-8087-46b1-b64d-e3f15811caf6/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:232 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/04996af222f21f33acf2dbb3b6b94b8be5ca45bb4627d0208b7720a427210892/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:372 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46ea9f9182e771ae4b2a53ab78cc99399a8f3eba9f63301332e3064c96a24c69/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d7b9f6a71d269f8ddf38e89179e4d4b9b8da09b774010f87c7d5614ddb1043f7/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5c702aef-eccc-4da1-8040-c860c607a466/volumes/kubernetes.io~projected/kube-api-access-p2vfb DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-371 DeviceMajor:0 DeviceMinor:371 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/54fd1c126cabbff9f892111cc0bbff2664cb51ea952375ab1eb54b8401f6cbf8/userdata/shm DeviceMajor:0 DeviceMinor:383 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aafb99e79b91dc43038e1c3d53e6955437495733c43e838913a87c6b2f17a553/userdata/shm DeviceMajor:0 DeviceMinor:390 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d/volumes/kubernetes.io~projected/kube-api-access-qjtk6 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/808521e6-a0b3-4a24-b0b0-5a3cabffe0f5/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a31aa40d-e199-4095-b9b9-f2c7e31822e2/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/593202543ae57c5e203fa3abb539eb711165e48b2bf2ba5cef1721e0bcc17287/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da5c17e9-1229-4d33-ba12-3eab4727e64d/volumes/kubernetes.io~projected/kube-api-access-rmtxv DeviceMajor:0 DeviceMinor:302 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21593b07-35ab-409e-98ef-74167147e27c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:277 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:308 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/950f0225-e1c9-4c9c-8fc6-25dba5db7120/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34905c4138e89ebabc77c38a62d4d277a3e9ad38e44c156e43ee57012e0deae7/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f1829b1-8087-46b1-b64d-e3f15811caf6/volumes/kubernetes.io~projected/kube-api-access-z27h9 DeviceMajor:0 DeviceMinor:237 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:309 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a/volumes/kubernetes.io~projected/kube-api-access-d92bb DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5c702aef-eccc-4da1-8040-c860c607a466/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/189b46e5-b36a-4ce4-be3e-b5486b7e45a8/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:216 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/da5c17e9-1229-4d33-ba12-3eab4727e64d/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:301 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:309 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7f1829b1-8087-46b1-b64d-e3f15811caf6/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:236 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8d43f954-119d-40c5-9727-1e25043da4ee/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:340 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2ae36e61-388d-4bc0-8810-aae71f3703c6/volumes/kubernetes.io~projected/kube-api-access-vwp8n DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dfad36cb6e1ed10a6cd6660f40a16c252c832f5e86b10f65345c745eaaae034d/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:312 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a23d62cf087a22dbe4e7bf329a60a92adba8cb51a46683861e9baba4d8d3c18b/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21593b07-35ab-409e-98ef-74167147e27c/volumes/kubernetes.io~projected/kube-api-access-lz24x DeviceMajor:0 DeviceMinor:283 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/da5c17e9-1229-4d33-ba12-3eab4727e64d/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:295 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:309 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8/volumes/kubernetes.io~projected/kube-api-access-krfth DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da5c17e9-1229-4d33-ba12-3eab4727e64d/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:299 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:317 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d43f954-119d-40c5-9727-1e25043da4ee/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:336 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/808521e6-a0b3-4a24-b0b0-5a3cabffe0f5/volumes/kubernetes.io~projected/kube-api-access-7ztrc DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/189b46e5-b36a-4ce4-be3e-b5486b7e45a8/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:215 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/253215521b57eb57279a0ac998f8cbcd172f5d06f7d99cad8392adf60220daeb/userdata/shm DeviceMajor:0 DeviceMinor:319 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/6841b7e6-42ec-493f-9613-ac4902212fac/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:315 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:309 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/183318eb-952e-4e93-b716-6ab50b9c58dd/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:373 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:310 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6841b7e6-42ec-493f-9613-ac4902212fac/volumes/kubernetes.io~projected/kube-api-access-2dpq2 DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~projected/kube-api-access-4ppdt DeviceMajor:0 DeviceMinor:318 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:313 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01498921daece598a7ce2734824ebc2fd4b2b973c4e826a4e09f9a58838d32b9/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:307 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6841b7e6-42ec-493f-9613-ac4902212fac/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:314 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/edbef9c655815b9877f936c73f1f75f3449440552d90723d5d0a8f2b9727a8da/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c3af705e-b3ba-4379-8214-07ba1b431824/volumes/kubernetes.io~projected/kube-api-access-8sm8p DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21593b07-35ab-409e-98ef-74167147e27c/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:282 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4067ec29eb4ea5cbff18adbbd3d2f6346dfa9ffdc48fbb0744ac13476c2fb181/userdata/shm DeviceMajor:0 DeviceMinor:303 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/828aaed34bc5f0464c0e592518791a70af36bb4a367070d703daf52da7d54d26/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/189b46e5-b36a-4ce4-be3e-b5486b7e45a8/volumes/kubernetes.io~projected/kube-api-access-sl65r DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/da5c17e9-1229-4d33-ba12-3eab4727e64d/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:300 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:311 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d43f954-119d-40c5-9727-1e25043da4ee/volumes/kubernetes.io~projected/kube-api-access-bczbc DeviceMajor:0 DeviceMinor:341 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5934c36a-2955-4f0c-a8e8-615f2fa07454/volumes/kubernetes.io~projected/kube-api-access-mx5l7 DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:316 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bcd8b205dfbf9b7c1e054114904f54e7451342bcfa11344e056753f200b7694b/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bb88c1ff-4b28-4812-9b2b-8876983720e5/volumes/kubernetes.io~projected/kube-api-access-qqcwl DeviceMajor:0 DeviceMinor:360 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78/volumes/kubernetes.io~projected/kube-api-access-4gstg DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f3721f94fe154516ba62cca3e2cbe1788b0776dd7701f70a95539931b316fef/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2fe33a545cb5bbcb29c98119be301ae623972d70eff09fbb658bc8dd3ecd78c5/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/acb30153b006b8ad51cb2725eeeb41d1406ddabeab22c41da892be6125a8d85a/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a31aa40d-e199-4095-b9b9-f2c7e31822e2/volumes/kubernetes.io~projected/kube-api-access-kqfx8 DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5510e59bc4d70b7949d7a6c736e5765e4da41eecb6bfbfed83b31498a7c5426/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c95fd8aec2140a47abfc68a4ebcb6b3b50eec9e9579bc892c31a5fff26eb8e7/userdata/shm DeviceMajor:0 DeviceMinor:387 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:253215521b57eb5 MacAddress:82:d3:3f:87:ca:40 Speed:10000 Mtu:8901} {Name:4067ec29eb4ea5c MacAddress:9a:02:2a:4a:2c:8e Speed:10000 Mtu:8901} {Name:54fd1c126cabbff MacAddress:fe:09:1a:02:85:5f Speed:10000 Mtu:8901} {Name:593202543ae57c5 MacAddress:fe:f5:95:e4:0a:6a Speed:10000 Mtu:8901} {Name:6c95fd8aec2140a MacAddress:0e:56:09:73:32:21 Speed:10000 Mtu:8901} {Name:828aaed34bc5f04 MacAddress:0e:69:2c:a6:c0:41 Speed:10000 Mtu:8901} {Name:a5510e59bc4d70b MacAddress:c6:5d:a7:e5:a9:78 Speed:10000 Mtu:8901} {Name:aafb99e79b91dc4 MacAddress:ee:6c:4e:6d:d5:79 Speed:10000 Mtu:8901} {Name:bcd8b205dfbf9b7 MacAddress:be:6a:d8:54:9c:78 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c1:09:2e:63 Speed:0 Mtu:9001} {Name:br-int MacAddress:7e:47:97:a1:a3:fb Speed:0 Mtu:8901} {Name:d7b9f6a71d269f8 MacAddress:a2:a1:27:70:0c:8c Speed:10000 Mtu:8901} {Name:dfad36cb6e1ed10 MacAddress:26:6d:1f:3f:df:b7 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c1:09:2e:63 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:12:52:63:fc:31:8b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:2e:62:88:c4:38:85 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:28:51.461997 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.461251 5801 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:28:51.461997 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.461295 5801 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:28:51.461997 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.461827 5801 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:28:51.462423 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.461859 5801 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-163.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:28:51.462423 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.462184 5801 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:28:51.462423 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.462198 5801 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:28:51.462423 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.462212 5801 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:28:51.462423 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.462238 5801 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:28:51.462423 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.462354 5801 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:28:51.462684 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.462617 5801 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:28:51.463294 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.463279 5801 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:28:51.463357 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.463326 5801 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:28:51.463357 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.463347 5801 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:28:51.463446 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.463361 5801 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:28:51.463446 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.463370 5801 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:28:51.463446 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.463379 5801 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:28:51.464217 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.464198 5801 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:28:51.464217 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.464214 5801 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:28:51.464691 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.464669 5801 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:28:51.467885 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.467871 5801 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:28:51.467885 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.467885 5801 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:28:51.467990 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.467980 5801 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 21:28:51.468177 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.468165 5801 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:28:51.468220 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.468204 5801 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:28:51.468306 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.468297 5801 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:28:51.468335 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.468312 5801 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:28:51.468335 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.468319 5801 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:28:51.468335 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.468324 5801 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:28:51.468335 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.468330 5801 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:28:51.468335 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.468336 5801 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:28:51.468471 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.468343 5801 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:28:51.468471 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.468352 5801 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:28:51.468471 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.468360 5801 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:28:51.468471 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.468365 5801 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:28:51.468471 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.468374 5801 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:28:51.468471 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.468383 5801 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:28:51.468471 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.468407 5801 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:28:51.468471 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.468412 5801 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:28:51.471900 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.471885 5801 server.go:1263] "Started kubelet" Oct 30 21:28:51.472370 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.471979 5801 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:28:51.472370 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.472039 5801 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:28:51.472433 ip-10-0-140-163 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:28:51.472731 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.472709 5801 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:28:51.473051 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.473016 5801 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:28:51.473836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.473818 5801 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:28:51.475688 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.475200 5801 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:28:51.475688 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.475220 5801 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:28:51.475688 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.475286 5801 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:22:41 +0000 UTC" deadline="2028-08-13 13:44:12.312786672 +0000 UTC" Oct 30 21:28:51.475688 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.475308 5801 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:28:51.475688 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.475318 5801 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15664h15m20.837471418s" Oct 30 21:28:51.475688 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.475327 5801 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:28:51.475688 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:28:51.475335 5801 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 21:28:51.475688 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.475496 5801 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:28:51.478609 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.478584 5801 factory.go:55] Registering systemd factory Oct 30 21:28:51.478820 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.478613 5801 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:28:51.480434 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.478859 5801 factory.go:153] Registering CRI-O factory Oct 30 21:28:51.480434 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.478870 5801 factory.go:223] Registration of the crio container factory successfully Oct 30 21:28:51.480434 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.478931 5801 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:28:51.480434 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.478952 5801 factory.go:103] Registering Raw factory Oct 30 21:28:51.480434 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.478968 5801 manager.go:1199] Started watching for new ooms in manager Oct 30 21:28:51.480434 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.479292 5801 manager.go:319] Starting recovery of all containers Oct 30 21:28:51.485072 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.484635 5801 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:28:51.485703 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.485681 5801 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:28:51.488136 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488090 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f1829b1-8087-46b1-b64d-e3f15811caf6" volumeName="kubernetes.io/secret/7f1829b1-8087-46b1-b64d-e3f15811caf6-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:28:51.488242 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488141 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" volumeName="kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 21:28:51.488242 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488157 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" volumeName="kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 21:28:51.488242 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488169 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5c17e9-1229-4d33-ba12-3eab4727e64d" volumeName="kubernetes.io/projected/da5c17e9-1229-4d33-ba12-3eab4727e64d-kube-api-access-rmtxv" seLinuxMountContext="" Oct 30 21:28:51.488242 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488180 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="189b46e5-b36a-4ce4-be3e-b5486b7e45a8" volumeName="kubernetes.io/projected/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-kube-api-access-sl65r" seLinuxMountContext="" Oct 30 21:28:51.488242 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488189 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21593b07-35ab-409e-98ef-74167147e27c" volumeName="kubernetes.io/configmap/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 21:28:51.488242 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488197 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f1829b1-8087-46b1-b64d-e3f15811caf6" volumeName="kubernetes.io/projected/7f1829b1-8087-46b1-b64d-e3f15811caf6-kube-api-access-z27h9" seLinuxMountContext="" Oct 30 21:28:51.488242 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488205 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" volumeName="kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 21:28:51.488242 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488215 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="189b46e5-b36a-4ce4-be3e-b5486b7e45a8" volumeName="kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-oauth-serving-cert" seLinuxMountContext="" Oct 30 21:28:51.488242 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488223 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21593b07-35ab-409e-98ef-74167147e27c" volumeName="kubernetes.io/secret/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-tls" seLinuxMountContext="" Oct 30 21:28:51.488242 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488231 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f1829b1-8087-46b1-b64d-e3f15811caf6" volumeName="kubernetes.io/secret/7f1829b1-8087-46b1-b64d-e3f15811caf6-prometheus-operator-tls" seLinuxMountContext="" Oct 30 21:28:51.488242 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488244 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d43f954-119d-40c5-9727-1e25043da4ee" volumeName="kubernetes.io/empty-dir/8d43f954-119d-40c5-9727-1e25043da4ee-tmp" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488257 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" volumeName="kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488270 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" volumeName="kubernetes.io/empty-dir/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-config-out" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488287 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d" volumeName="kubernetes.io/projected/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-kube-api-access-qjtk6" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488299 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="189b46e5-b36a-4ce4-be3e-b5486b7e45a8" volumeName="kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-service-ca" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488307 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="189b46e5-b36a-4ce4-be3e-b5486b7e45a8" volumeName="kubernetes.io/secret/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-serving-cert" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488335 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="846a43d3-aaa8-4d0f-bc4f-32916a05fb9a" volumeName="kubernetes.io/empty-dir/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-tuned" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488348 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d43f954-119d-40c5-9727-1e25043da4ee" volumeName="kubernetes.io/secret/8d43f954-119d-40c5-9727-1e25043da4ee-console-oauth-config" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488360 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5c17e9-1229-4d33-ba12-3eab4727e64d" volumeName="kubernetes.io/secret/da5c17e9-1229-4d33-ba12-3eab4727e64d-federate-client-tls" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488372 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5934c36a-2955-4f0c-a8e8-615f2fa07454" volumeName="kubernetes.io/projected/5934c36a-2955-4f0c-a8e8-615f2fa07454-kube-api-access-mx5l7" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488384 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8" volumeName="kubernetes.io/projected/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-kube-api-access-krfth" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488396 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5c17e9-1229-4d33-ba12-3eab4727e64d" volumeName="kubernetes.io/secret/da5c17e9-1229-4d33-ba12-3eab4727e64d-secret-telemeter-client" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488405 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c702aef-eccc-4da1-8040-c860c607a466" volumeName="kubernetes.io/empty-dir/5c702aef-eccc-4da1-8040-c860c607a466-tmp-dir" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488412 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c702aef-eccc-4da1-8040-c860c607a466" volumeName="kubernetes.io/projected/5c702aef-eccc-4da1-8040-c860c607a466-kube-api-access-p2vfb" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488420 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c702aef-eccc-4da1-8040-c860c607a466" volumeName="kubernetes.io/secret/5c702aef-eccc-4da1-8040-c860c607a466-metrics-tls" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488427 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6841b7e6-42ec-493f-9613-ac4902212fac" volumeName="kubernetes.io/projected/6841b7e6-42ec-493f-9613-ac4902212fac-kube-api-access-2dpq2" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488435 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f1829b1-8087-46b1-b64d-e3f15811caf6" volumeName="kubernetes.io/configmap/7f1829b1-8087-46b1-b64d-e3f15811caf6-metrics-client-ca" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488446 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5c17e9-1229-4d33-ba12-3eab4727e64d" volumeName="kubernetes.io/secret/da5c17e9-1229-4d33-ba12-3eab4727e64d-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488454 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21593b07-35ab-409e-98ef-74167147e27c" volumeName="kubernetes.io/empty-dir/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-textfile" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488463 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="846a43d3-aaa8-4d0f-bc4f-32916a05fb9a" volumeName="kubernetes.io/projected/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-kube-api-access-d92bb" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488470 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="950f0225-e1c9-4c9c-8fc6-25dba5db7120" volumeName="kubernetes.io/secret/950f0225-e1c9-4c9c-8fc6-25dba5db7120-agent-certs" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488477 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" volumeName="kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-web-config" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488488 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb88c1ff-4b28-4812-9b2b-8876983720e5" volumeName="kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488500 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4" volumeName="kubernetes.io/projected/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-kube-api-access-2hgqm" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488509 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4" volumeName="kubernetes.io/configmap/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488517 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="183318eb-952e-4e93-b716-6ab50b9c58dd" volumeName="kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488524 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d43f954-119d-40c5-9727-1e25043da4ee" volumeName="kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-oauth-serving-cert" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488538 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" volumeName="kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488548 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" volumeName="kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488560 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" volumeName="kubernetes.io/projected/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-kube-api-access-4ppdt" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488569 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a31aa40d-e199-4095-b9b9-f2c7e31822e2" volumeName="kubernetes.io/configmap/a31aa40d-e199-4095-b9b9-f2c7e31822e2-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488582 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3af705e-b3ba-4379-8214-07ba1b431824" volumeName="kubernetes.io/projected/c3af705e-b3ba-4379-8214-07ba1b431824-kube-api-access-8sm8p" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488590 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5c17e9-1229-4d33-ba12-3eab4727e64d" volumeName="kubernetes.io/configmap/da5c17e9-1229-4d33-ba12-3eab4727e64d-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488601 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d43f954-119d-40c5-9727-1e25043da4ee" volumeName="kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-console-config" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488612 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d43f954-119d-40c5-9727-1e25043da4ee" volumeName="kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-service-ca" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488625 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" volumeName="kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488638 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" volumeName="kubernetes.io/empty-dir/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-db" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488650 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" volumeName="kubernetes.io/projected/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-tls-assets" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488662 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" volumeName="kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488674 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ae36e61-388d-4bc0-8810-aae71f3703c6" volumeName="kubernetes.io/configmap/2ae36e61-388d-4bc0-8810-aae71f3703c6-iptables-alerter-script" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488682 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" volumeName="kubernetes.io/projected/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-kube-api-access-4gstg" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488689 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8" volumeName="kubernetes.io/configmap/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-cni-binary-copy" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488698 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="950f0225-e1c9-4c9c-8fc6-25dba5db7120" volumeName="kubernetes.io/configmap/950f0225-e1c9-4c9c-8fc6-25dba5db7120-konnectivity-ca" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488709 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" volumeName="kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488717 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" volumeName="kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-config" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488732 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5c17e9-1229-4d33-ba12-3eab4727e64d" volumeName="kubernetes.io/configmap/da5c17e9-1229-4d33-ba12-3eab4727e64d-telemeter-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488744 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="189b46e5-b36a-4ce4-be3e-b5486b7e45a8" volumeName="kubernetes.io/secret/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-oauth-config" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488770 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21593b07-35ab-409e-98ef-74167147e27c" volumeName="kubernetes.io/configmap/21593b07-35ab-409e-98ef-74167147e27c-metrics-client-ca" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488781 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c702aef-eccc-4da1-8040-c860c607a466" volumeName="kubernetes.io/configmap/5c702aef-eccc-4da1-8040-c860c607a466-config-volume" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488789 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6841b7e6-42ec-493f-9613-ac4902212fac" volumeName="kubernetes.io/configmap/6841b7e6-42ec-493f-9613-ac4902212fac-ovnkube-config" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488796 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6841b7e6-42ec-493f-9613-ac4902212fac" volumeName="kubernetes.io/configmap/6841b7e6-42ec-493f-9613-ac4902212fac-env-overrides" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488803 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d43f954-119d-40c5-9727-1e25043da4ee" volumeName="kubernetes.io/secret/8d43f954-119d-40c5-9727-1e25043da4ee-console-serving-cert" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488811 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" volumeName="kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488823 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" volumeName="kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488837 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78" volumeName="kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs" seLinuxMountContext="" Oct 30 21:28:51.488808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488850 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6841b7e6-42ec-493f-9613-ac4902212fac" volumeName="kubernetes.io/secret/6841b7e6-42ec-493f-9613-ac4902212fac-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488862 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="808521e6-a0b3-4a24-b0b0-5a3cabffe0f5" volumeName="kubernetes.io/projected/808521e6-a0b3-4a24-b0b0-5a3cabffe0f5-kube-api-access-7ztrc" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488875 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="808521e6-a0b3-4a24-b0b0-5a3cabffe0f5" volumeName="kubernetes.io/secret/808521e6-a0b3-4a24-b0b0-5a3cabffe0f5-cert" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488887 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a31aa40d-e199-4095-b9b9-f2c7e31822e2" volumeName="kubernetes.io/projected/a31aa40d-e199-4095-b9b9-f2c7e31822e2-kube-api-access-kqfx8" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488899 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4" volumeName="kubernetes.io/configmap/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488911 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5c17e9-1229-4d33-ba12-3eab4727e64d" volumeName="kubernetes.io/secret/da5c17e9-1229-4d33-ba12-3eab4727e64d-telemeter-client-tls" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488923 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="846a43d3-aaa8-4d0f-bc4f-32916a05fb9a" volumeName="kubernetes.io/empty-dir/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-tmp" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488935 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21593b07-35ab-409e-98ef-74167147e27c" volumeName="kubernetes.io/secret/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488947 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5934c36a-2955-4f0c-a8e8-615f2fa07454" volumeName="kubernetes.io/configmap/5934c36a-2955-4f0c-a8e8-615f2fa07454-serviceca" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488959 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8" volumeName="kubernetes.io/configmap/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-multus-daemon-config" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488971 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" volumeName="kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-grpc-tls" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488983 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a31aa40d-e199-4095-b9b9-f2c7e31822e2" volumeName="kubernetes.io/secret/a31aa40d-e199-4095-b9b9-f2c7e31822e2-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.488996 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21593b07-35ab-409e-98ef-74167147e27c" volumeName="kubernetes.io/projected/21593b07-35ab-409e-98ef-74167147e27c-kube-api-access-lz24x" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.489008 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" volumeName="kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.489019 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4" volumeName="kubernetes.io/configmap/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-cni-binary-copy" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.489030 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ae36e61-388d-4bc0-8810-aae71f3703c6" volumeName="kubernetes.io/projected/2ae36e61-388d-4bc0-8810-aae71f3703c6-kube-api-access-vwp8n" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.489041 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d43f954-119d-40c5-9727-1e25043da4ee" volumeName="kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.489050 5801 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.489053 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d43f954-119d-40c5-9727-1e25043da4ee" volumeName="kubernetes.io/projected/8d43f954-119d-40c5-9727-1e25043da4ee-kube-api-access-bczbc" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.489066 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" volumeName="kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.489078 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a31aa40d-e199-4095-b9b9-f2c7e31822e2" volumeName="kubernetes.io/empty-dir/a31aa40d-e199-4095-b9b9-f2c7e31822e2-data-volume" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.489090 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3af705e-b3ba-4379-8214-07ba1b431824" volumeName="kubernetes.io/empty-dir/c3af705e-b3ba-4379-8214-07ba1b431824-tmp-dir" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.489101 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5c17e9-1229-4d33-ba12-3eab4727e64d" volumeName="kubernetes.io/configmap/da5c17e9-1229-4d33-ba12-3eab4727e64d-metrics-client-ca" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.489114 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="189b46e5-b36a-4ce4-be3e-b5486b7e45a8" volumeName="kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-config" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.489125 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="189b46e5-b36a-4ce4-be3e-b5486b7e45a8" volumeName="kubernetes.io/empty-dir/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-tmp" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.489136 5801 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6841b7e6-42ec-493f-9613-ac4902212fac" volumeName="kubernetes.io/configmap/6841b7e6-42ec-493f-9613-ac4902212fac-ovnkube-script-lib" seLinuxMountContext="" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.489147 5801 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:28:51.491184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.489155 5801 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:28:51.497876 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.497851 5801 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 21:28:51.514656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.514611 5801 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:28:51.516180 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.516154 5801 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:28:51.516309 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.516186 5801 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:28:51.516309 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.516211 5801 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:28:51.516309 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:28:51.516260 5801 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:28:51.521838 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.521142 5801 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:28:51.527341 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.527299 5801 generic.go:356] "Generic (PLEG): container finished" podID="cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4" containerID="794f49925c3ae47119ef384cb7faea61a69f9a39e498f2deb87727553d4b6477" exitCode=0 Oct 30 21:28:51.527341 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.527336 5801 generic.go:356] "Generic (PLEG): container finished" podID="cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4" containerID="4c55c9fac74c10a75c547a362a820f150b8d5a5796bccaeb9a3252d592c0d800" exitCode=0 Oct 30 21:28:51.527499 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.527347 5801 generic.go:356] "Generic (PLEG): container finished" podID="cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4" containerID="41f6bac12e157d04e12e48956ae478191f187cd5d3125e78d1debd21784dae77" exitCode=0 Oct 30 21:28:51.527499 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.527356 5801 generic.go:356] "Generic (PLEG): container finished" podID="cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4" containerID="6d58dd436714e080ad79a776dccae59c9cb254adae1ff1f60ce897d56c9154e6" exitCode=0 Oct 30 21:28:51.527499 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.527364 5801 generic.go:356] "Generic (PLEG): container finished" podID="cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4" containerID="b30bd30e30ec76492ef051e7c965b9db53d6897a56304b806a5e42db46c3233c" exitCode=0 Oct 30 21:28:51.527499 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.527371 5801 generic.go:356] "Generic (PLEG): container finished" podID="cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4" containerID="2e524e22ab8a5f1c5fbd8662c4aa5447e439efd936a6720704a4163d72f7edb9" exitCode=0 Oct 30 21:28:51.531150 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.531122 5801 generic.go:356] "Generic (PLEG): container finished" podID="21593b07-35ab-409e-98ef-74167147e27c" containerID="c091ea83c74aad3ea2fe30942815804a025d15ae29391539061b22d15600e580" exitCode=0 Oct 30 21:28:51.541103 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.541081 5801 generic.go:356] "Generic (PLEG): container finished" podID="40b5c76fae82f6d7a38a1a745d5d3947" containerID="15d831d327b6d46836a138b6aa732eecca3a25e6333f65eada26bf0d1f5095c0" exitCode=0 Oct 30 21:28:51.557738 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.557704 5801 generic.go:356] "Generic (PLEG): container finished" podID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerID="ef6880b249d4139651e75607e731fc966d7a838c977325ba312cb6be6292545c" exitCode=0 Oct 30 21:28:51.586518 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.586495 5801 manager.go:324] Recovery completed Oct 30 21:28:51.610293 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.610225 5801 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:28:51.610293 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.610243 5801 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:28:51.610293 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.610261 5801 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:28:51.610469 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.610385 5801 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 21:28:51.610469 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.610395 5801 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 21:28:51.610532 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.610505 5801 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 21:28:51.610532 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.610516 5801 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 21:28:51.610532 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.610522 5801 policy_none.go:50] "Start" Oct 30 21:28:51.610616 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.610547 5801 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:28:51.610616 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.610558 5801 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:28:51.610696 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.610685 5801 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 21:28:51.610725 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.610696 5801 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 21:28:51.610725 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.610702 5801 policy_none.go:44] "Start" Oct 30 21:28:51.615181 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.615156 5801 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:28:51.617210 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:28:51.617174 5801 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 21:28:51.617976 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.617962 5801 manager.go:341] "Starting Device Plugin manager" Oct 30 21:28:51.618437 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:28:51.617990 5801 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:28:51.618437 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.618433 5801 server.go:87] "Starting device plugin registration server" Oct 30 21:28:51.618706 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.618692 5801 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:28:51.618786 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.618704 5801 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:28:51.618852 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.618821 5801 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:28:51.618965 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.618930 5801 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:28:51.619016 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.618975 5801 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:28:51.619016 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.618982 5801 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:28:51.619114 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.619035 5801 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:28:51.618962087Z","UUID":"e1144ccc-a014-4f93-a18f-61c067f820cc","Handler":null,"Name":"","Endpoint":""} Oct 30 21:28:51.620898 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:28:51.620735 5801 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:28:51.621195 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.621178 5801 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:28:51.621332 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.621314 5801 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:28:51.719731 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.719688 5801 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:28:51.721792 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.721772 5801 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:28:51.721908 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.721809 5801 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:28:51.721908 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.721824 5801 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:28:51.721908 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.721878 5801 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:28:51.738529 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.738497 5801 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:28:51.738641 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.738589 5801 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:28:51.818155 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.818089 5801 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal"] Oct 30 21:28:51.818686 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.818645 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" event={"ID":"40b5c76fae82f6d7a38a1a745d5d3947","Type":"ContainerStarted","Data":"a6acde858f50a052f287c1205b925236d7ad4dd9752fb2b0fbd2b0424676fad5"} Oct 30 21:28:51.818782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.818699 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" event={"ID":"40b5c76fae82f6d7a38a1a745d5d3947","Type":"ContainerDied","Data":"15d831d327b6d46836a138b6aa732eecca3a25e6333f65eada26bf0d1f5095c0"} Oct 30 21:28:51.818782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.818718 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" event={"ID":"40b5c76fae82f6d7a38a1a745d5d3947","Type":"ContainerStarted","Data":"a23d62cf087a22dbe4e7bf329a60a92adba8cb51a46683861e9baba4d8d3c18b"} Oct 30 21:28:51.818782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.818731 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" event={"ID":"3b404058373133d538844156a64ac398","Type":"ContainerStarted","Data":"4c876390172c7f6471923b80845c336ad6f3a26b2ef16272050dcf639ef92a58"} Oct 30 21:28:51.818782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.818743 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" event={"ID":"3b404058373133d538844156a64ac398","Type":"ContainerStarted","Data":"edbef9c655815b9877f936c73f1f75f3449440552d90723d5d0a8f2b9727a8da"} Oct 30 21:28:51.818782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.818746 5801 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:28:51.818937 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.818925 5801 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:28:51.839416 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.839399 5801 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:28:51.839469 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:28:51.839436 5801 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:28:51.840172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.840162 5801 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:28:51.840208 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:28:51.840183 5801 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:28:51.899566 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.899510 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40b5c76fae82f6d7a38a1a745d5d3947-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"40b5c76fae82f6d7a38a1a745d5d3947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:28:51.899566 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.899540 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40b5c76fae82f6d7a38a1a745d5d3947-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"40b5c76fae82f6d7a38a1a745d5d3947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:28:51.899566 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:51.899558 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b404058373133d538844156a64ac398-config\") pod \"kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"3b404058373133d538844156a64ac398\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:28:52.000185 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.000146 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40b5c76fae82f6d7a38a1a745d5d3947-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"40b5c76fae82f6d7a38a1a745d5d3947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:28:52.000294 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.000164 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40b5c76fae82f6d7a38a1a745d5d3947-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"40b5c76fae82f6d7a38a1a745d5d3947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:28:52.000294 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.000215 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40b5c76fae82f6d7a38a1a745d5d3947-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"40b5c76fae82f6d7a38a1a745d5d3947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:28:52.000294 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.000234 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b404058373133d538844156a64ac398-config\") pod \"kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"3b404058373133d538844156a64ac398\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:28:52.000294 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.000269 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b404058373133d538844156a64ac398-config\") pod \"kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"3b404058373133d538844156a64ac398\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:28:52.000294 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.000287 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40b5c76fae82f6d7a38a1a745d5d3947-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"40b5c76fae82f6d7a38a1a745d5d3947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:28:52.463861 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.463826 5801 apiserver.go:52] "Watching apiserver" Oct 30 21:28:52.475119 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.475093 5801 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:28:52.475707 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.475677 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-7d6nt","openshift-network-diagnostics/network-check-target-kv4cs","openshift-console/console-5c5fdfbf9b-l7clr","openshift-multus/multus-additional-cni-plugins-vcn8r","openshift-multus/multus-hl8qs","kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal","openshift-dns/node-resolver-j2j9f","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal","openshift-console/console-54ddb8d6b-4rlwf","openshift-ingress-canary/ingress-canary-wdr8m","kube-system/global-pull-secret-syncer-c2vrg","openshift-image-registry/node-ca-db7tz","openshift-dns/dns-default-99chm","openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc","openshift-monitoring/telemeter-client-6d84c6fb95-spv7p","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l","openshift-insights/insights-runtime-extractor-htw5p","openshift-cluster-node-tuning-operator/tuned-6fx47","openshift-ovn-kubernetes/ovnkube-node-xh6hf","openshift-monitoring/prometheus-k8s-0","openshift-network-operator/iptables-alerter-wd8lr","openshift-multus/network-metrics-daemon-4r4zb","openshift-monitoring/node-exporter-h847d"] Oct 30 21:28:52.494270 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.494244 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2pcx6\"" Oct 30 21:28:52.494814 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.494798 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:28:52.495917 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.495903 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-z8x7v\"" Oct 30 21:28:52.496106 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.496095 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d996q\"" Oct 30 21:28:52.496245 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.496230 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g2c5m\"" Oct 30 21:28:52.496302 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.496289 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pqqf8\"" Oct 30 21:28:52.496332 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.496301 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:28:52.496332 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.496294 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:28:52.499871 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.499857 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:28:52.499908 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.499860 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wxqgz\"" Oct 30 21:28:52.500268 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.500254 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cjjj4\"" Oct 30 21:28:52.500418 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.500397 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-99k6q\"" Oct 30 21:28:52.500418 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.500414 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:28:52.500553 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.500424 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qcj4q\"" Oct 30 21:28:52.500553 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.500428 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ct8zc\"" Oct 30 21:28:52.500553 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.500436 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-f4rkh\"" Oct 30 21:28:52.500553 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.500458 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:28:52.500553 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.500489 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rpdb2\"" Oct 30 21:28:52.504805 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.504790 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 21:28:52.505105 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.505090 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 21:28:52.508126 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.508109 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 21:28:52.508206 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.508138 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 21:28:52.508317 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.508302 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 21:28:52.508380 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.508308 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 21:28:52.508380 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.508348 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 21:28:52.508478 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.508393 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:28:52.508478 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.508472 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:28:52.508580 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.508562 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 21:28:52.508833 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.508816 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 21:28:52.508960 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.508846 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-flwfj\"" Oct 30 21:28:52.508960 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.508864 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 21:28:52.509102 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.509087 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 21:28:52.509155 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.509129 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 21:28:52.509155 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.509133 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 21:28:52.509155 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.509140 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:28:52.509267 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.509255 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 21:28:52.509804 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.509793 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 21:28:52.509863 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.509852 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:28:52.509900 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.509889 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:28:52.509931 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.509905 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:28:52.509931 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.509915 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7mnf0orvh0m0j\"" Oct 30 21:28:52.509931 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.509919 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:28:52.510372 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.510359 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:28:52.510423 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.510410 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:28:52.510458 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.510421 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2wf4q\"" Oct 30 21:28:52.510458 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.510425 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:28:52.510526 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.510482 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:28:52.510526 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.510490 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:28:52.510526 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.510517 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:28:52.510622 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.510561 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:28:52.510622 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.510578 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:28:52.510622 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.510596 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:28:52.510760 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.510624 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:28:52.510760 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.510642 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 21:28:52.510760 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.510709 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:28:52.511420 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.511408 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-r96r5\"" Oct 30 21:28:52.511469 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.511457 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dj9gj\"" Oct 30 21:28:52.511950 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.511937 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 21:28:52.511989 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.511968 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 21:28:52.512121 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.512106 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 21:28:52.512157 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.512120 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:28:52.512157 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.512130 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 21:28:52.513617 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.513602 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:28:52.513671 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.513624 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:28:52.513671 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.513655 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 21:28:52.513788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.513658 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:28:52.513788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.513680 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:28:52.513788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.513691 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:28:52.513788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.513684 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 21:28:52.513788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.513670 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:28:52.513788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.513721 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:28:52.513788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.513703 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-qhqpw\"" Oct 30 21:28:52.513788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.513710 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:28:52.513788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.513713 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:28:52.513788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.513731 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:28:52.514070 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.513794 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9vdcb\"" Oct 30 21:28:52.514266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.514250 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:28:52.514323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.514250 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:28:52.514323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.514277 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:28:52.514323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.514280 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 21:28:52.514323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.514256 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:28:52.514323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.514277 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:28:52.514891 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.514874 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vf75f\"" Oct 30 21:28:52.514958 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.514889 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:28:52.514958 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.514895 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 21:28:52.514958 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.514909 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:28:52.514958 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.514915 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 21:28:52.515102 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.515082 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-75pfq\"" Oct 30 21:28:52.515991 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.515976 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:28:52.516071 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.515988 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:28:52.516071 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.516023 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 21:28:52.516071 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.516037 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 21:28:52.516450 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.516435 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:28:52.518094 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.518080 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:28:52.523119 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.523072 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 21:28:52.523940 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.523918 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 21:28:52.524279 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.524260 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 21:28:52.525489 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.525462 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 21:28:52.576718 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.576691 5801 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:28:52.576917 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.576795 5801 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:28:52.577590 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.577573 5801 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:28:52.602134 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602101 5801 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:28:52.602258 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:28:52.602154 5801 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:28:52.602316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602301 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.602348 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602334 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-config-out\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.602401 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602364 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-run-netns\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.602401 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602391 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2ae36e61-388d-4bc0-8810-aae71f3703c6-iptables-alerter-script\") pod \"iptables-alerter-wd8lr\" (UID: \"2ae36e61-388d-4bc0-8810-aae71f3703c6\") " pod="openshift-network-operator/iptables-alerter-wd8lr" Oct 30 21:28:52.602484 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602415 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-service-ca\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:52.602484 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602460 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-config-out\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.602571 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602441 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bczbc\" (UniqueName: \"kubernetes.io/projected/8d43f954-119d-40c5-9727-1e25043da4ee-kube-api-access-bczbc\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:52.602571 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602535 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.602571 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602542 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-run-netns\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.602571 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602567 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a31aa40d-e199-4095-b9b9-f2c7e31822e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:52.602782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602582 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.602782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602601 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-web-config\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.602782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602629 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-var-lib-kubelet\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.602782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602651 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d43f954-119d-40c5-9727-1e25043da4ee-tmp\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:52.602782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602675 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-log-socket\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.602782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602700 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/da5c17e9-1229-4d33-ba12-3eab4727e64d-federate-client-tls\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:52.602782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602729 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-service-ca\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:52.602782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602739 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2ae36e61-388d-4bc0-8810-aae71f3703c6-iptables-alerter-script\") pod \"iptables-alerter-wd8lr\" (UID: \"2ae36e61-388d-4bc0-8810-aae71f3703c6\") " pod="openshift-network-operator/iptables-alerter-wd8lr" Oct 30 21:28:52.603109 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602729 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da5c17e9-1229-4d33-ba12-3eab4727e64d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:52.603109 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602814 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d43f954-119d-40c5-9727-1e25043da4ee-tmp\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:52.603109 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602813 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.603109 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602839 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a31aa40d-e199-4095-b9b9-f2c7e31822e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:52.603109 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602979 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da5c17e9-1229-4d33-ba12-3eab4727e64d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:52.603109 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.602984 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-sysctl-conf\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.603109 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603046 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d92bb\" (UniqueName: \"kubernetes.io/projected/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-kube-api-access-d92bb\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.603109 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603057 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/da5c17e9-1229-4d33-ba12-3eab4727e64d-federate-client-tls\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:52.603109 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603084 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-web-config\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.603439 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603112 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-host\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.603439 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603139 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-tuned\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.603439 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603156 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-multus-cni-dir\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.603439 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603231 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/183318eb-952e-4e93-b716-6ab50b9c58dd-kubelet-config\") pod \"global-pull-secret-syncer-c2vrg\" (UID: \"183318eb-952e-4e93-b716-6ab50b9c58dd\") " pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:52.603439 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603237 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-tuned\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.603439 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603265 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-sysctl-d\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.603439 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603310 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.603439 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603330 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-var-lib-cni-bin\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.603439 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603347 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwp8n\" (UniqueName: \"kubernetes.io/projected/2ae36e61-388d-4bc0-8810-aae71f3703c6-kube-api-access-vwp8n\") pod \"iptables-alerter-wd8lr\" (UID: \"2ae36e61-388d-4bc0-8810-aae71f3703c6\") " pod="openshift-network-operator/iptables-alerter-wd8lr" Oct 30 21:28:52.603439 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603362 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-etc-openvswitch\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.603439 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603377 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-run-ovn-kubernetes\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.603896 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603527 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6841b7e6-42ec-493f-9613-ac4902212fac-ovn-node-metrics-cert\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.603896 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603553 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3af705e-b3ba-4379-8214-07ba1b431824-tmp-dir\") pod \"node-resolver-j2j9f\" (UID: \"c3af705e-b3ba-4379-8214-07ba1b431824\") " pod="openshift-dns/node-resolver-j2j9f" Oct 30 21:28:52.603896 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603600 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3af705e-b3ba-4379-8214-07ba1b431824-tmp-dir\") pod \"node-resolver-j2j9f\" (UID: \"c3af705e-b3ba-4379-8214-07ba1b431824\") " pod="openshift-dns/node-resolver-j2j9f" Oct 30 21:28:52.603896 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603679 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:28:52.603896 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603717 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a31aa40d-e199-4095-b9b9-f2c7e31822e2-data-volume\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:52.603896 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603735 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.603896 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603746 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a31aa40d-e199-4095-b9b9-f2c7e31822e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:52.603896 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603791 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-config\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.603896 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603815 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.603896 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603820 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a31aa40d-e199-4095-b9b9-f2c7e31822e2-data-volume\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:52.603896 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603792 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6841b7e6-42ec-493f-9613-ac4902212fac-ovn-node-metrics-cert\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.603896 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603838 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:28:52.603896 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603840 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.603896 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603878 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.603896 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603896 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21593b07-35ab-409e-98ef-74167147e27c-metrics-client-ca\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603912 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-kubelet\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603933 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-systemd-units\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603950 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-slash\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.603992 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-run-ovn\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604011 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a31aa40d-e199-4095-b9b9-f2c7e31822e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604029 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2vfb\" (UniqueName: \"kubernetes.io/projected/5c702aef-eccc-4da1-8040-c860c607a466-kube-api-access-p2vfb\") pod \"dns-default-99chm\" (UID: \"5c702aef-eccc-4da1-8040-c860c607a466\") " pod="openshift-dns/dns-default-99chm" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604003 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-config\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604042 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604049 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-textfile\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604069 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604071 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz24x\" (UniqueName: \"kubernetes.io/projected/21593b07-35ab-409e-98ef-74167147e27c-kube-api-access-lz24x\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604108 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5c17e9-1229-4d33-ba12-3eab4727e64d-serving-certs-ca-bundle\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604121 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-textfile\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604134 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-run\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604161 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604173 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21593b07-35ab-409e-98ef-74167147e27c-metrics-client-ca\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604188 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-system-cni-dir\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604292 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret\") pod \"global-pull-secret-syncer-c2vrg\" (UID: \"183318eb-952e-4e93-b716-6ab50b9c58dd\") " pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604291 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5c17e9-1229-4d33-ba12-3eab4727e64d-serving-certs-ca-bundle\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604310 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-service-ca\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604319 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604337 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604357 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6841b7e6-42ec-493f-9613-ac4902212fac-ovnkube-config\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604372 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-systemd\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604388 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqcwl\" (UniqueName: \"kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl\") pod \"network-check-target-kv4cs\" (UID: \"bb88c1ff-4b28-4812-9b2b-8876983720e5\") " pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604393 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604403 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-device-dir\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604423 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-accelerators-collector-config\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604467 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-service-ca\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604517 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5c17e9-1229-4d33-ba12-3eab4727e64d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:52.604543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604555 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx5l7\" (UniqueName: \"kubernetes.io/projected/5934c36a-2955-4f0c-a8e8-615f2fa07454-kube-api-access-mx5l7\") pod \"node-ca-db7tz\" (UID: \"5934c36a-2955-4f0c-a8e8-615f2fa07454\") " pod="openshift-image-registry/node-ca-db7tz" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604579 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-run-systemd\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604607 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6841b7e6-42ec-493f-9613-ac4902212fac-env-overrides\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604608 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-accelerators-collector-config\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604608 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/183318eb-952e-4e93-b716-6ab50b9c58dd-original-pull-secret\") pod \"global-pull-secret-syncer-c2vrg\" (UID: \"183318eb-952e-4e93-b716-6ab50b9c58dd\") " pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604632 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c702aef-eccc-4da1-8040-c860c607a466-tmp-dir\") pod \"dns-default-99chm\" (UID: \"5c702aef-eccc-4da1-8040-c860c607a466\") " pod="openshift-dns/dns-default-99chm" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604646 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6841b7e6-42ec-493f-9613-ac4902212fac-ovnkube-config\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604697 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c702aef-eccc-4da1-8040-c860c607a466-tmp-dir\") pod \"dns-default-99chm\" (UID: \"5c702aef-eccc-4da1-8040-c860c607a466\") " pod="openshift-dns/dns-default-99chm" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604746 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604784 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-cnibin\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604793 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6841b7e6-42ec-493f-9613-ac4902212fac-env-overrides\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604833 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-hostroot\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604866 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjtk6\" (UniqueName: \"kubernetes.io/projected/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-kube-api-access-qjtk6\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604884 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-node-log\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604904 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/950f0225-e1c9-4c9c-8fc6-25dba5db7120-agent-certs\") pod \"konnectivity-agent-7d6nt\" (UID: \"950f0225-e1c9-4c9c-8fc6-25dba5db7120\") " pod="kube-system/konnectivity-agent-7d6nt" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604919 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-cnibin\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604933 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-run-openvswitch\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604949 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-wtmp\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604965 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5c17e9-1229-4d33-ba12-3eab4727e64d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.604995 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605028 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-system-cni-dir\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605154 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605157 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605170 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/950f0225-e1c9-4c9c-8fc6-25dba5db7120-agent-certs\") pod \"konnectivity-agent-7d6nt\" (UID: \"950f0225-e1c9-4c9c-8fc6-25dba5db7120\") " pod="kube-system/konnectivity-agent-7d6nt" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605203 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605223 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/183318eb-952e-4e93-b716-6ab50b9c58dd-dbus\") pod \"global-pull-secret-syncer-c2vrg\" (UID: \"183318eb-952e-4e93-b716-6ab50b9c58dd\") " pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605239 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-oauth-serving-cert\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605253 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21593b07-35ab-409e-98ef-74167147e27c-sys\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605308 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605330 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-kubernetes\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605347 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqfx8\" (UniqueName: \"kubernetes.io/projected/a31aa40d-e199-4095-b9b9-f2c7e31822e2-kube-api-access-kqfx8\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605359 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605364 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-tmp\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605418 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-oauth-serving-cert\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605431 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-tmp\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605474 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/21593b07-35ab-409e-98ef-74167147e27c-root\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605507 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-modprobe-d\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605546 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-sys\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605550 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.605570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605584 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-tmp\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605627 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gstg\" (UniqueName: \"kubernetes.io/projected/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-kube-api-access-4gstg\") pod \"network-metrics-daemon-4r4zb\" (UID: \"3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78\") " pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605680 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z27h9\" (UniqueName: \"kubernetes.io/projected/7f1829b1-8087-46b1-b64d-e3f15811caf6-kube-api-access-z27h9\") pod \"prometheus-operator-9cd89b74c-xh9wc\" (UID: \"7f1829b1-8087-46b1-b64d-e3f15811caf6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605692 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-tmp\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605700 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605736 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f1829b1-8087-46b1-b64d-e3f15811caf6-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xh9wc\" (UID: \"7f1829b1-8087-46b1-b64d-e3f15811caf6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605810 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sm8p\" (UniqueName: \"kubernetes.io/projected/c3af705e-b3ba-4379-8214-07ba1b431824-kube-api-access-8sm8p\") pod \"node-resolver-j2j9f\" (UID: \"c3af705e-b3ba-4379-8214-07ba1b431824\") " pod="openshift-dns/node-resolver-j2j9f" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605838 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605940 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605972 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f1829b1-8087-46b1-b64d-e3f15811caf6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xh9wc\" (UID: \"7f1829b1-8087-46b1-b64d-e3f15811caf6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.605996 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-console-config\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606027 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl65r\" (UniqueName: \"kubernetes.io/projected/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-kube-api-access-sl65r\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606031 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606050 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f1829b1-8087-46b1-b64d-e3f15811caf6-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xh9wc\" (UID: \"7f1829b1-8087-46b1-b64d-e3f15811caf6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606053 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-var-lib-kubelet\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606075 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606099 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-run-k8s-cni-cncf-io\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606148 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606149 5801 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606172 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f1829b1-8087-46b1-b64d-e3f15811caf6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xh9wc\" (UID: \"7f1829b1-8087-46b1-b64d-e3f15811caf6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606198 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dpq2\" (UniqueName: \"kubernetes.io/projected/6841b7e6-42ec-493f-9613-ac4902212fac-kube-api-access-2dpq2\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:28:52.606209 5801 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606234 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmtxv\" (UniqueName: \"kubernetes.io/projected/da5c17e9-1229-4d33-ba12-3eab4727e64d-kube-api-access-rmtxv\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606266 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hgqm\" (UniqueName: \"kubernetes.io/projected/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-kube-api-access-2hgqm\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606284 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-lib-modules\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606305 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606325 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-multus-conf-dir\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606338 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-console-config\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606401 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606443 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c702aef-eccc-4da1-8040-c860c607a466-metrics-tls\") pod \"dns-default-99chm\" (UID: \"5c702aef-eccc-4da1-8040-c860c607a466\") " pod="openshift-dns/dns-default-99chm" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606467 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-cni-bin\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606491 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c3af705e-b3ba-4379-8214-07ba1b431824-hosts-file\") pod \"node-resolver-j2j9f\" (UID: \"c3af705e-b3ba-4379-8214-07ba1b431824\") " pod="openshift-dns/node-resolver-j2j9f" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606513 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f1829b1-8087-46b1-b64d-e3f15811caf6-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xh9wc\" (UID: \"7f1829b1-8087-46b1-b64d-e3f15811caf6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606551 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-config\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606620 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da5c17e9-1229-4d33-ba12-3eab4727e64d-metrics-client-ca\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606723 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c702aef-eccc-4da1-8040-c860c607a466-metrics-tls\") pod \"dns-default-99chm\" (UID: \"5c702aef-eccc-4da1-8040-c860c607a466\") " pod="openshift-dns/dns-default-99chm" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606771 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da5c17e9-1229-4d33-ba12-3eab4727e64d-metrics-client-ca\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606776 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-sysconfig\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606804 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606805 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f1829b1-8087-46b1-b64d-e3f15811caf6-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xh9wc\" (UID: \"7f1829b1-8087-46b1-b64d-e3f15811caf6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606827 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-cni-binary-copy\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606866 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-multus-socket-dir-parent\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606894 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-etc-kubernetes\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606921 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-registration-dir\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606949 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-tls\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606967 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.606978 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5934c36a-2955-4f0c-a8e8-615f2fa07454-serviceca\") pod \"node-ca-db7tz\" (UID: \"5934c36a-2955-4f0c-a8e8-615f2fa07454\") " pod="openshift-image-registry/node-ca-db7tz" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607012 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-var-lib-cni-multus\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607032 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-cni-binary-copy\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607041 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-sys-fs\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607072 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d43f954-119d-40c5-9727-1e25043da4ee-console-oauth-config\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607099 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6841b7e6-42ec-493f-9613-ac4902212fac-ovnkube-script-lib\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607126 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-serving-cert\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607153 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-oauth-serving-cert\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607176 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5c702aef-eccc-4da1-8040-c860c607a466-config-volume\") pod \"dns-default-99chm\" (UID: \"5c702aef-eccc-4da1-8040-c860c607a466\") " pod="openshift-dns/dns-default-99chm" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607184 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d43f954-119d-40c5-9727-1e25043da4ee-console-oauth-config\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607203 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:28:52.607205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607182 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-tls\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607256 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-config\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607277 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607303 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-multus-daemon-config\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607329 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-run-multus-certs\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607354 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a31aa40d-e199-4095-b9b9-f2c7e31822e2-crio-socket\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607405 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5c702aef-eccc-4da1-8040-c860c607a466-config-volume\") pod \"dns-default-99chm\" (UID: \"5c702aef-eccc-4da1-8040-c860c607a466\") " pod="openshift-dns/dns-default-99chm" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607425 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ppdt\" (UniqueName: \"kubernetes.io/projected/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-kube-api-access-4ppdt\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607454 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607481 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-serving-cert\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607571 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-oauth-serving-cert\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607608 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5934c36a-2955-4f0c-a8e8-615f2fa07454-serviceca\") pod \"node-ca-db7tz\" (UID: \"5934c36a-2955-4f0c-a8e8-615f2fa07454\") " pod="openshift-image-registry/node-ca-db7tz" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607484 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607642 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-var-lib-openvswitch\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607661 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-cni-netd\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607686 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607686 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/da5c17e9-1229-4d33-ba12-3eab4727e64d-secret-telemeter-client\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607724 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/da5c17e9-1229-4d33-ba12-3eab4727e64d-telemeter-client-tls\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607702 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607872 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-multus-daemon-config\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607874 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6841b7e6-42ec-493f-9613-ac4902212fac-ovnkube-script-lib\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607882 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5934c36a-2955-4f0c-a8e8-615f2fa07454-host\") pod \"node-ca-db7tz\" (UID: \"5934c36a-2955-4f0c-a8e8-615f2fa07454\") " pod="openshift-image-registry/node-ca-db7tz" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.607972 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-socket-dir\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.608003 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ae36e61-388d-4bc0-8810-aae71f3703c6-host-slash\") pod \"iptables-alerter-wd8lr\" (UID: \"2ae36e61-388d-4bc0-8810-aae71f3703c6\") " pod="openshift-network-operator/iptables-alerter-wd8lr" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.608058 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d43f954-119d-40c5-9727-1e25043da4ee-console-serving-cert\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.608088 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/da5c17e9-1229-4d33-ba12-3eab4727e64d-secret-telemeter-client\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.608113 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-trusted-ca-bundle\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.608087 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/da5c17e9-1229-4d33-ba12-3eab4727e64d-telemeter-client-tls\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.608143 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.608256 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-oauth-config\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.608282 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d43f954-119d-40c5-9727-1e25043da4ee-console-serving-cert\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.608291 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/950f0225-e1c9-4c9c-8fc6-25dba5db7120-konnectivity-ca\") pod \"konnectivity-agent-7d6nt\" (UID: \"950f0225-e1c9-4c9c-8fc6-25dba5db7120\") " pod="kube-system/konnectivity-agent-7d6nt" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.608325 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs\") pod \"network-metrics-daemon-4r4zb\" (UID: \"3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78\") " pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.608412 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.608461 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/950f0225-e1c9-4c9c-8fc6-25dba5db7120-konnectivity-ca\") pod \"konnectivity-agent-7d6nt\" (UID: \"950f0225-e1c9-4c9c-8fc6-25dba5db7120\") " pod="kube-system/konnectivity-agent-7d6nt" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.608485 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-oauth-config\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.608486 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krfth\" (UniqueName: \"kubernetes.io/projected/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-kube-api-access-krfth\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.608528 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-metrics-certs\") pod \"network-metrics-daemon-4r4zb\" (UID: \"3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78\") " pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.608610 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/808521e6-a0b3-4a24-b0b0-5a3cabffe0f5-cert\") pod \"ingress-canary-wdr8m\" (UID: \"808521e6-a0b3-4a24-b0b0-5a3cabffe0f5\") " pod="openshift-ingress-canary/ingress-canary-wdr8m" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.608892 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ztrc\" (UniqueName: \"kubernetes.io/projected/808521e6-a0b3-4a24-b0b0-5a3cabffe0f5-kube-api-access-7ztrc\") pod \"ingress-canary-wdr8m\" (UID: \"808521e6-a0b3-4a24-b0b0-5a3cabffe0f5\") " pod="openshift-ingress-canary/ingress-canary-wdr8m" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.609176 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/808521e6-a0b3-4a24-b0b0-5a3cabffe0f5-cert\") pod \"ingress-canary-wdr8m\" (UID: \"808521e6-a0b3-4a24-b0b0-5a3cabffe0f5\") " pod="openshift-ingress-canary/ingress-canary-wdr8m" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.609186 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-trusted-ca-bundle\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:52.610079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.609740 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.624290 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.624267 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bczbc\" (UniqueName: \"kubernetes.io/projected/8d43f954-119d-40c5-9727-1e25043da4ee-kube-api-access-bczbc\") pod \"console-5c5fdfbf9b-l7clr\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:52.638486 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.638452 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwp8n\" (UniqueName: \"kubernetes.io/projected/2ae36e61-388d-4bc0-8810-aae71f3703c6-kube-api-access-vwp8n\") pod \"iptables-alerter-wd8lr\" (UID: \"2ae36e61-388d-4bc0-8810-aae71f3703c6\") " pod="openshift-network-operator/iptables-alerter-wd8lr" Oct 30 21:28:52.641388 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.641371 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz24x\" (UniqueName: \"kubernetes.io/projected/21593b07-35ab-409e-98ef-74167147e27c-kube-api-access-lz24x\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:52.644977 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.644956 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z27h9\" (UniqueName: \"kubernetes.io/projected/7f1829b1-8087-46b1-b64d-e3f15811caf6-kube-api-access-z27h9\") pod \"prometheus-operator-9cd89b74c-xh9wc\" (UID: \"7f1829b1-8087-46b1-b64d-e3f15811caf6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xh9wc" Oct 30 21:28:52.645471 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.645451 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sm8p\" (UniqueName: \"kubernetes.io/projected/c3af705e-b3ba-4379-8214-07ba1b431824-kube-api-access-8sm8p\") pod \"node-resolver-j2j9f\" (UID: \"c3af705e-b3ba-4379-8214-07ba1b431824\") " pod="openshift-dns/node-resolver-j2j9f" Oct 30 21:28:52.645509 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.645464 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjtk6\" (UniqueName: \"kubernetes.io/projected/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-kube-api-access-qjtk6\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:28:52.645539 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.645506 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gstg\" (UniqueName: \"kubernetes.io/projected/3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78-kube-api-access-4gstg\") pod \"network-metrics-daemon-4r4zb\" (UID: \"3d8eebdf-2dfa-4039-8bb1-ca648e0a8d78\") " pod="openshift-multus/network-metrics-daemon-4r4zb" Oct 30 21:28:52.646327 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.646308 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d92bb\" (UniqueName: \"kubernetes.io/projected/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-kube-api-access-d92bb\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.646387 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.646333 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx5l7\" (UniqueName: \"kubernetes.io/projected/5934c36a-2955-4f0c-a8e8-615f2fa07454-kube-api-access-mx5l7\") pod \"node-ca-db7tz\" (UID: \"5934c36a-2955-4f0c-a8e8-615f2fa07454\") " pod="openshift-image-registry/node-ca-db7tz" Oct 30 21:28:52.646387 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.646338 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqfx8\" (UniqueName: \"kubernetes.io/projected/a31aa40d-e199-4095-b9b9-f2c7e31822e2-kube-api-access-kqfx8\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:52.648589 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.648569 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqcwl\" (UniqueName: \"kubernetes.io/projected/bb88c1ff-4b28-4812-9b2b-8876983720e5-kube-api-access-qqcwl\") pod \"network-check-target-kv4cs\" (UID: \"bb88c1ff-4b28-4812-9b2b-8876983720e5\") " pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:28:52.650153 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.650137 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl65r\" (UniqueName: \"kubernetes.io/projected/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-kube-api-access-sl65r\") pod \"console-54ddb8d6b-4rlwf\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:52.652339 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.652323 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2vfb\" (UniqueName: \"kubernetes.io/projected/5c702aef-eccc-4da1-8040-c860c607a466-kube-api-access-p2vfb\") pod \"dns-default-99chm\" (UID: \"5c702aef-eccc-4da1-8040-c860c607a466\") " pod="openshift-dns/dns-default-99chm" Oct 30 21:28:52.652588 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.652570 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dpq2\" (UniqueName: \"kubernetes.io/projected/6841b7e6-42ec-493f-9613-ac4902212fac-kube-api-access-2dpq2\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.653403 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.653384 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ppdt\" (UniqueName: \"kubernetes.io/projected/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-kube-api-access-4ppdt\") pod \"prometheus-k8s-0\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:52.653686 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.653667 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hgqm\" (UniqueName: \"kubernetes.io/projected/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-kube-api-access-2hgqm\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:28:52.655271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.655251 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krfth\" (UniqueName: \"kubernetes.io/projected/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-kube-api-access-krfth\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.655868 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.655850 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmtxv\" (UniqueName: \"kubernetes.io/projected/da5c17e9-1229-4d33-ba12-3eab4727e64d-kube-api-access-rmtxv\") pod \"telemeter-client-6d84c6fb95-spv7p\" (UID: \"da5c17e9-1229-4d33-ba12-3eab4727e64d\") " pod="openshift-monitoring/telemeter-client-6d84c6fb95-spv7p" Oct 30 21:28:52.660054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.660036 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ztrc\" (UniqueName: \"kubernetes.io/projected/808521e6-a0b3-4a24-b0b0-5a3cabffe0f5-kube-api-access-7ztrc\") pod \"ingress-canary-wdr8m\" (UID: \"808521e6-a0b3-4a24-b0b0-5a3cabffe0f5\") " pod="openshift-ingress-canary/ingress-canary-wdr8m" Oct 30 21:28:52.709250 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709222 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-sysctl-d\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.709250 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709253 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-var-lib-cni-bin\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.709468 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709270 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-etc-openvswitch\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.709468 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709286 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-run-ovn-kubernetes\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.709468 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709340 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-run-ovn-kubernetes\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.709468 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709340 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-var-lib-cni-bin\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.709468 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709360 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-kubelet\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.709468 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709360 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-etc-openvswitch\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.709468 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709385 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-systemd-units\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.709468 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709403 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-slash\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.709468 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709420 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-run-ovn\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.709468 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709419 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-sysctl-d\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.709468 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709420 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-kubelet\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.709468 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709426 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-systemd-units\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.709468 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709455 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-run-ovn\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.709468 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709471 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-slash\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709492 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-run\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709516 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-system-cni-dir\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709537 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-run\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709540 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-systemd\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709564 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-device-dir\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709586 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-system-cni-dir\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709589 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-run-systemd\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709617 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-cnibin\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709618 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-run-systemd\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709634 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-systemd\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709640 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-device-dir\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709641 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-hostroot\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709661 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-cnibin\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709690 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-node-log\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709716 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-cnibin\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709741 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-node-log\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709795 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-run-openvswitch\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709806 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-hostroot\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709818 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-wtmp\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709822 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-cnibin\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709845 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-run-openvswitch\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709885 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-system-cni-dir\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709924 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/21593b07-35ab-409e-98ef-74167147e27c-node-exporter-wtmp\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709927 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-system-cni-dir\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709939 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/183318eb-952e-4e93-b716-6ab50b9c58dd-dbus\") pod \"global-pull-secret-syncer-c2vrg\" (UID: \"183318eb-952e-4e93-b716-6ab50b9c58dd\") " pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709957 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21593b07-35ab-409e-98ef-74167147e27c-sys\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709972 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.709988 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-kubernetes\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710003 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/21593b07-35ab-409e-98ef-74167147e27c-root\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710013 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21593b07-35ab-409e-98ef-74167147e27c-sys\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710016 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-modprobe-d\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710060 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/21593b07-35ab-409e-98ef-74167147e27c-root\") pod \"node-exporter-h847d\" (UID: \"21593b07-35ab-409e-98ef-74167147e27c\") " pod="openshift-monitoring/node-exporter-h847d" Oct 30 21:28:52.710047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710063 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vcn8r\" (UID: \"cf15b3a1-2cbb-4eb2-8ac0-7bdf39fc72c4\") " pod="openshift-multus/multus-additional-cni-plugins-vcn8r" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710040 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-sys\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710060 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-kubernetes\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710063 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/183318eb-952e-4e93-b716-6ab50b9c58dd-dbus\") pod \"global-pull-secret-syncer-c2vrg\" (UID: \"183318eb-952e-4e93-b716-6ab50b9c58dd\") " pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710107 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-var-lib-kubelet\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710120 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-sys\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710122 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-modprobe-d\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710130 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-run-k8s-cni-cncf-io\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710155 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-run-k8s-cni-cncf-io\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710163 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-var-lib-kubelet\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710194 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-lib-modules\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710214 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-multus-conf-dir\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710228 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-cni-bin\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710241 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c3af705e-b3ba-4379-8214-07ba1b431824-hosts-file\") pod \"node-resolver-j2j9f\" (UID: \"c3af705e-b3ba-4379-8214-07ba1b431824\") " pod="openshift-dns/node-resolver-j2j9f" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710252 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-lib-modules\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710257 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-sysconfig\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710279 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-multus-socket-dir-parent\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710283 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-sysconfig\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710293 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-etc-kubernetes\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710310 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-cni-bin\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710314 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-registration-dir\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710329 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c3af705e-b3ba-4379-8214-07ba1b431824-hosts-file\") pod \"node-resolver-j2j9f\" (UID: \"c3af705e-b3ba-4379-8214-07ba1b431824\") " pod="openshift-dns/node-resolver-j2j9f" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710333 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-multus-conf-dir\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710341 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-multus-socket-dir-parent\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710354 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-var-lib-cni-multus\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710341 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-etc-kubernetes\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710367 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-registration-dir\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710380 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-sys-fs\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710409 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-var-lib-cni-multus\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710409 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-run-multus-certs\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710434 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a31aa40d-e199-4095-b9b9-f2c7e31822e2-crio-socket\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710442 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-run-multus-certs\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710448 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710468 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710486 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-sys-fs\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710495 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a31aa40d-e199-4095-b9b9-f2c7e31822e2-crio-socket\") pod \"insights-runtime-extractor-htw5p\" (UID: \"a31aa40d-e199-4095-b9b9-f2c7e31822e2\") " pod="openshift-insights/insights-runtime-extractor-htw5p" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710502 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710499 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-var-lib-openvswitch\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710522 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710538 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-var-lib-openvswitch\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710572 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-cni-netd\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710604 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-cni-netd\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710604 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5934c36a-2955-4f0c-a8e8-615f2fa07454-host\") pod \"node-ca-db7tz\" (UID: \"5934c36a-2955-4f0c-a8e8-615f2fa07454\") " pod="openshift-image-registry/node-ca-db7tz" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710628 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-socket-dir\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710645 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ae36e61-388d-4bc0-8810-aae71f3703c6-host-slash\") pod \"iptables-alerter-wd8lr\" (UID: \"2ae36e61-388d-4bc0-8810-aae71f3703c6\") " pod="openshift-network-operator/iptables-alerter-wd8lr" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710666 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710696 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-run-netns\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710705 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d-socket-dir\") pod \"aws-ebs-csi-driver-node-v747l\" (UID: \"f0b83dac-47b9-4fb6-b39b-5e2dd2850c2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v747l" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710715 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ae36e61-388d-4bc0-8810-aae71f3703c6-host-slash\") pod \"iptables-alerter-wd8lr\" (UID: \"2ae36e61-388d-4bc0-8810-aae71f3703c6\") " pod="openshift-network-operator/iptables-alerter-wd8lr" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710716 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710735 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-run-netns\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710741 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-run-netns\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710775 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-var-lib-kubelet\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710730 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5934c36a-2955-4f0c-a8e8-615f2fa07454-host\") pod \"node-ca-db7tz\" (UID: \"5934c36a-2955-4f0c-a8e8-615f2fa07454\") " pod="openshift-image-registry/node-ca-db7tz" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710797 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-log-socket\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710802 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-host-run-netns\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710813 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-host-var-lib-kubelet\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710814 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-sysctl-conf\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710830 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6841b7e6-42ec-493f-9613-ac4902212fac-log-socket\") pod \"ovnkube-node-xh6hf\" (UID: \"6841b7e6-42ec-493f-9613-ac4902212fac\") " pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710836 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-host\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710852 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-multus-cni-dir\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710859 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-host\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710867 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/183318eb-952e-4e93-b716-6ab50b9c58dd-kubelet-config\") pod \"global-pull-secret-syncer-c2vrg\" (UID: \"183318eb-952e-4e93-b716-6ab50b9c58dd\") " pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710896 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8-multus-cni-dir\") pod \"multus-hl8qs\" (UID: \"5ba16fc2-45ad-4ae0-b077-b4fc6bd7dda8\") " pod="openshift-multus/multus-hl8qs" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710903 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/846a43d3-aaa8-4d0f-bc4f-32916a05fb9a-etc-sysctl-conf\") pod \"tuned-6fx47\" (UID: \"846a43d3-aaa8-4d0f-bc4f-32916a05fb9a\") " pod="openshift-cluster-node-tuning-operator/tuned-6fx47" Oct 30 21:28:52.711161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.710912 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/183318eb-952e-4e93-b716-6ab50b9c58dd-kubelet-config\") pod \"global-pull-secret-syncer-c2vrg\" (UID: \"183318eb-952e-4e93-b716-6ab50b9c58dd\") " pod="kube-system/global-pull-secret-syncer-c2vrg" Oct 30 21:28:52.754883 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:52.754797 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54ddb8d6b-4rlwf"] Oct 30 21:28:53.277085 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:53.277050 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:53.386355 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:53.386321 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:53.996146 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:53.996105 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:54.000837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:54.000814 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:54.095373 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:54.095346 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-99chm" Oct 30 21:28:54.666073 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:54.666033 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:54.777023 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:54.776982 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:55.376250 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:55.376214 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:55.887345 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:55.887311 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:55.887547 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:55.887534 5801 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:28:55.957046 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:55.957007 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:55.975037 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:55.975002 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:28:56.483379 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:56.483343 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:56.629461 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:56.629422 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:57.567063 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:57.567029 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:28:57.913921 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:57.913828 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:28:58.109359 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:58.109323 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:58.291368 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:58.291293 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xh6hf" Oct 30 21:28:59.124396 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:59.124368 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:28:59.805194 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:59.805154 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:59.899013 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:28:59.898979 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:00.276573 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:29:00.276493 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:29:02.816951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:02.816856 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:02.915515 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:02.915474 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:07.918575 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:07.918542 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:29:08.167965 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:08.167926 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:08.277412 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:08.277335 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:08.524340 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:08.524301 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:08.618505 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:08.618477 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:09.309488 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:09.309462 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7d6nt" Oct 30 21:29:09.668696 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:09.668613 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:09.770240 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:09.770213 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:09.930839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:09.930747 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:11.405212 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.405177 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:29:11.405661 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.405490 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="prometheus" containerID="cri-o://2a248c86f433a34a28abbdd7cf5827290268d64646a26957352516bb4d73ddf2" gracePeriod=600 Oct 30 21:29:11.405661 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.405540 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="kube-rbac-proxy" containerID="cri-o://923bee7254814f53592fe25939cea6f23dc928ef5cd2b2a84c8f28cbf46bdbc8" gracePeriod=600 Oct 30 21:29:11.405661 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.405573 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="thanos-sidecar" containerID="cri-o://bfb255833c61bfe2ed287a07c34622eaff10f9eea540a4764fedbce5957d847d" gracePeriod=600 Oct 30 21:29:11.405661 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.405574 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="config-reloader" containerID="cri-o://0ccb9b005019e336e5c868e7111c6f5a0c67848f0425b23b2f41ff4a5efd58fe" gracePeriod=600 Oct 30 21:29:11.405661 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.405584 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="kube-rbac-proxy-web" containerID="cri-o://ca6b1295e7c5546f7289353bb4dc679654a0dafa29957bfa0a98be9ae2a9c130" gracePeriod=600 Oct 30 21:29:11.405970 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.405686 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://13a723438766d7bad387bff6a969b504d00ed2613d5b714084603e5a5842599f" gracePeriod=600 Oct 30 21:29:11.641632 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.641603 5801 generic.go:356] "Generic (PLEG): container finished" podID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerID="13a723438766d7bad387bff6a969b504d00ed2613d5b714084603e5a5842599f" exitCode=0 Oct 30 21:29:11.641632 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.641624 5801 generic.go:356] "Generic (PLEG): container finished" podID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerID="923bee7254814f53592fe25939cea6f23dc928ef5cd2b2a84c8f28cbf46bdbc8" exitCode=0 Oct 30 21:29:11.641632 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.641631 5801 generic.go:356] "Generic (PLEG): container finished" podID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerID="ca6b1295e7c5546f7289353bb4dc679654a0dafa29957bfa0a98be9ae2a9c130" exitCode=0 Oct 30 21:29:11.641632 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.641636 5801 generic.go:356] "Generic (PLEG): container finished" podID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerID="bfb255833c61bfe2ed287a07c34622eaff10f9eea540a4764fedbce5957d847d" exitCode=0 Oct 30 21:29:11.641983 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.641641 5801 generic.go:356] "Generic (PLEG): container finished" podID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerID="0ccb9b005019e336e5c868e7111c6f5a0c67848f0425b23b2f41ff4a5efd58fe" exitCode=0 Oct 30 21:29:11.641983 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.641648 5801 generic.go:356] "Generic (PLEG): container finished" podID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerID="2a248c86f433a34a28abbdd7cf5827290268d64646a26957352516bb4d73ddf2" exitCode=0 Oct 30 21:29:11.641983 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.641677 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"999c5bdd-e3fc-4103-9d65-0d3a228fc78b","Type":"ContainerDied","Data":"13a723438766d7bad387bff6a969b504d00ed2613d5b714084603e5a5842599f"} Oct 30 21:29:11.641983 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.641782 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"999c5bdd-e3fc-4103-9d65-0d3a228fc78b","Type":"ContainerDied","Data":"923bee7254814f53592fe25939cea6f23dc928ef5cd2b2a84c8f28cbf46bdbc8"} Oct 30 21:29:11.641983 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.641823 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"999c5bdd-e3fc-4103-9d65-0d3a228fc78b","Type":"ContainerDied","Data":"ca6b1295e7c5546f7289353bb4dc679654a0dafa29957bfa0a98be9ae2a9c130"} Oct 30 21:29:11.641983 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.641864 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"999c5bdd-e3fc-4103-9d65-0d3a228fc78b","Type":"ContainerDied","Data":"bfb255833c61bfe2ed287a07c34622eaff10f9eea540a4764fedbce5957d847d"} Oct 30 21:29:11.641983 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.641905 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"999c5bdd-e3fc-4103-9d65-0d3a228fc78b","Type":"ContainerDied","Data":"0ccb9b005019e336e5c868e7111c6f5a0c67848f0425b23b2f41ff4a5efd58fe"} Oct 30 21:29:11.641983 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.641945 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"999c5bdd-e3fc-4103-9d65-0d3a228fc78b","Type":"ContainerDied","Data":"2a248c86f433a34a28abbdd7cf5827290268d64646a26957352516bb4d73ddf2"} Oct 30 21:29:11.660252 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.660203 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:11.771564 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.771531 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-kube-rbac-proxy\") pod \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " Oct 30 21:29:11.771564 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.771567 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-web-config\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-web-config\") pod \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " Oct 30 21:29:11.771817 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.771587 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-grpc-tls\") pod \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " Oct 30 21:29:11.771817 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.771604 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-config\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-config\") pod \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " Oct 30 21:29:11.771817 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.771628 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-serving-certs-ca-bundle\") pod \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " Oct 30 21:29:11.771817 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.771798 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-0\") pod \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " Oct 30 21:29:11.772035 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.771870 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-kube-api-access-4ppdt\" (UniqueName: \"kubernetes.io/projected/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-kube-api-access-4ppdt\") pod \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " Oct 30 21:29:11.772035 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.771899 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-tls\") pod \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " Oct 30 21:29:11.772035 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.771935 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-2\") pod \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " Oct 30 21:29:11.772035 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.771970 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " Oct 30 21:29:11.772035 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.772000 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-metrics-client-ca\") pod \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " Oct 30 21:29:11.772294 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.772040 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-kubelet-serving-ca-bundle\") pod \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " Oct 30 21:29:11.772294 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.772071 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-trusted-ca-bundle\") pod \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " Oct 30 21:29:11.772294 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.772152 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-tls-assets\" (UniqueName: \"kubernetes.io/projected/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-tls-assets\") pod \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " Oct 30 21:29:11.772294 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.772181 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-thanos-prometheus-http-client-file\") pod \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " Oct 30 21:29:11.772294 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.772215 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-config-out\" (UniqueName: \"kubernetes.io/empty-dir/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-config-out\") pod \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " Oct 30 21:29:11.772294 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.772251 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-1\") pod \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " Oct 30 21:29:11.772294 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.772282 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " Oct 30 21:29:11.772653 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.772316 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-metrics-client-certs\") pod \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " Oct 30 21:29:11.772653 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.772346 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-db\") pod \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\" (UID: \"999c5bdd-e3fc-4103-9d65-0d3a228fc78b\") " Oct 30 21:29:11.775339 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.772032 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-serving-certs-ca-bundle" pod "999c5bdd-e3fc-4103-9d65-0d3a228fc78b" (UID: "999c5bdd-e3fc-4103-9d65-0d3a228fc78b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:11.775339 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.775259 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-grpc-tls" pod "999c5bdd-e3fc-4103-9d65-0d3a228fc78b" (UID: "999c5bdd-e3fc-4103-9d65-0d3a228fc78b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:11.775339 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.772392 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-2" pod "999c5bdd-e3fc-4103-9d65-0d3a228fc78b" (UID: "999c5bdd-e3fc-4103-9d65-0d3a228fc78b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:11.775339 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.772881 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-0" pod "999c5bdd-e3fc-4103-9d65-0d3a228fc78b" (UID: "999c5bdd-e3fc-4103-9d65-0d3a228fc78b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:11.775339 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.773019 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-db" pod "999c5bdd-e3fc-4103-9d65-0d3a228fc78b" (UID: "999c5bdd-e3fc-4103-9d65-0d3a228fc78b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:29:11.775339 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.774348 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-1" pod "999c5bdd-e3fc-4103-9d65-0d3a228fc78b" (UID: "999c5bdd-e3fc-4103-9d65-0d3a228fc78b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:11.775339 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.774464 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-kubelet-serving-ca-bundle" pod "999c5bdd-e3fc-4103-9d65-0d3a228fc78b" (UID: "999c5bdd-e3fc-4103-9d65-0d3a228fc78b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:11.775339 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.774737 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-metrics-client-ca" pod "999c5bdd-e3fc-4103-9d65-0d3a228fc78b" (UID: "999c5bdd-e3fc-4103-9d65-0d3a228fc78b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:11.775339 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.775133 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-kube-rbac-proxy" pod "999c5bdd-e3fc-4103-9d65-0d3a228fc78b" (UID: "999c5bdd-e3fc-4103-9d65-0d3a228fc78b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:11.776003 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.775512 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-kube-api-access-4ppdt" pod "999c5bdd-e3fc-4103-9d65-0d3a228fc78b" (UID: "999c5bdd-e3fc-4103-9d65-0d3a228fc78b"). InnerVolumeSpecName "kube-api-access-4ppdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:29:11.776003 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.775549 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-config" pod "999c5bdd-e3fc-4103-9d65-0d3a228fc78b" (UID: "999c5bdd-e3fc-4103-9d65-0d3a228fc78b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:11.776003 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.775609 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-trusted-ca-bundle" pod "999c5bdd-e3fc-4103-9d65-0d3a228fc78b" (UID: "999c5bdd-e3fc-4103-9d65-0d3a228fc78b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:11.776003 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.775659 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-thanos-sidecar-tls" pod "999c5bdd-e3fc-4103-9d65-0d3a228fc78b" (UID: "999c5bdd-e3fc-4103-9d65-0d3a228fc78b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:11.776003 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.775920 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-tls-assets" pod "999c5bdd-e3fc-4103-9d65-0d3a228fc78b" (UID: "999c5bdd-e3fc-4103-9d65-0d3a228fc78b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:29:11.776389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.776366 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-tls" pod "999c5bdd-e3fc-4103-9d65-0d3a228fc78b" (UID: "999c5bdd-e3fc-4103-9d65-0d3a228fc78b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:11.777055 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.777037 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-kube-rbac-proxy-web" pod "999c5bdd-e3fc-4103-9d65-0d3a228fc78b" (UID: "999c5bdd-e3fc-4103-9d65-0d3a228fc78b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:11.777384 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.777366 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-thanos-prometheus-http-client-file" pod "999c5bdd-e3fc-4103-9d65-0d3a228fc78b" (UID: "999c5bdd-e3fc-4103-9d65-0d3a228fc78b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:11.777531 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.777512 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-metrics-client-certs" pod "999c5bdd-e3fc-4103-9d65-0d3a228fc78b" (UID: "999c5bdd-e3fc-4103-9d65-0d3a228fc78b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:11.777673 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.777656 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-config-out" pod "999c5bdd-e3fc-4103-9d65-0d3a228fc78b" (UID: "999c5bdd-e3fc-4103-9d65-0d3a228fc78b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:29:11.785114 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.785094 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-web-config" pod "999c5bdd-e3fc-4103-9d65-0d3a228fc78b" (UID: "999c5bdd-e3fc-4103-9d65-0d3a228fc78b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:11.856343 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.856312 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:11.873360 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.873328 5801 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:11.873360 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.873361 5801 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:11.873466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.873372 5801 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-metrics-client-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:11.873466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.873383 5801 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-db\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:11.873466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.873391 5801 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-kube-rbac-proxy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:11.873466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.873400 5801 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-web-config\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:11.873466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.873409 5801 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-grpc-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:11.873466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.873417 5801 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-config\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:11.873466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.873425 5801 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:11.873466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.873434 5801 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:11.873466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.873442 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ppdt\" (UniqueName: \"kubernetes.io/projected/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-kube-api-access-4ppdt\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:11.873466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.873450 5801 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:11.873466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.873459 5801 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:11.873466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.873469 5801 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:11.873846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.873480 5801 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-metrics-client-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:11.873846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.873489 5801 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:11.873846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.873498 5801 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:11.873846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.873506 5801 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-tls-assets\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:11.873846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.873515 5801 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:11.873846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.873525 5801 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/999c5bdd-e3fc-4103-9d65-0d3a228fc78b-config-out\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:11.955636 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:11.955570 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:12.651316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.651280 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"999c5bdd-e3fc-4103-9d65-0d3a228fc78b","Type":"ContainerDied","Data":"253215521b57eb57279a0ac998f8cbcd172f5d06f7d99cad8392adf60220daeb"} Oct 30 21:29:12.651718 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.651339 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.651718 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.651376 5801 scope.go:122] "RemoveContainer" containerID="13a723438766d7bad387bff6a969b504d00ed2613d5b714084603e5a5842599f" Oct 30 21:29:12.660101 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.660082 5801 scope.go:122] "RemoveContainer" containerID="923bee7254814f53592fe25939cea6f23dc928ef5cd2b2a84c8f28cbf46bdbc8" Oct 30 21:29:12.666618 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.666599 5801 scope.go:122] "RemoveContainer" containerID="ca6b1295e7c5546f7289353bb4dc679654a0dafa29957bfa0a98be9ae2a9c130" Oct 30 21:29:12.673423 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.673407 5801 scope.go:122] "RemoveContainer" containerID="bfb255833c61bfe2ed287a07c34622eaff10f9eea540a4764fedbce5957d847d" Oct 30 21:29:12.675416 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.675397 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:29:12.680926 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.680353 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:29:12.680926 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.680633 5801 scope.go:122] "RemoveContainer" containerID="0ccb9b005019e336e5c868e7111c6f5a0c67848f0425b23b2f41ff4a5efd58fe" Oct 30 21:29:12.687160 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.687145 5801 scope.go:122] "RemoveContainer" containerID="2a248c86f433a34a28abbdd7cf5827290268d64646a26957352516bb4d73ddf2" Oct 30 21:29:12.696109 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.696081 5801 scope.go:122] "RemoveContainer" containerID="ef6880b249d4139651e75607e731fc966d7a838c977325ba312cb6be6292545c" Oct 30 21:29:12.711709 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.711613 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:29:12.711941 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.711927 5801 cpu_manager.go:414] "containerMap: removing container" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="init-config-reloader" Oct 30 21:29:12.711994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.711942 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="init-config-reloader" Oct 30 21:29:12.711994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.711955 5801 cpu_manager.go:414] "containerMap: removing container" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="prometheus" Oct 30 21:29:12.711994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.711959 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="prometheus" Oct 30 21:29:12.711994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.711968 5801 cpu_manager.go:414] "containerMap: removing container" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="kube-rbac-proxy-web" Oct 30 21:29:12.711994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.711973 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="kube-rbac-proxy-web" Oct 30 21:29:12.711994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.711981 5801 cpu_manager.go:414] "containerMap: removing container" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="thanos-sidecar" Oct 30 21:29:12.711994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.711987 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="thanos-sidecar" Oct 30 21:29:12.711994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.711992 5801 cpu_manager.go:414] "containerMap: removing container" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="config-reloader" Oct 30 21:29:12.711994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.711997 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="config-reloader" Oct 30 21:29:12.712305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.712010 5801 cpu_manager.go:414] "containerMap: removing container" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="kube-rbac-proxy-thanos" Oct 30 21:29:12.712305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.712016 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="kube-rbac-proxy-thanos" Oct 30 21:29:12.712305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.712025 5801 cpu_manager.go:414] "containerMap: removing container" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="kube-rbac-proxy" Oct 30 21:29:12.712305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.712029 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="kube-rbac-proxy" Oct 30 21:29:12.712305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.712066 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="prometheus" Oct 30 21:29:12.712305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.712073 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="kube-rbac-proxy-web" Oct 30 21:29:12.712305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.712078 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="thanos-sidecar" Oct 30 21:29:12.712305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.712083 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="config-reloader" Oct 30 21:29:12.712305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.712090 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="kube-rbac-proxy-thanos" Oct 30 21:29:12.712305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.712096 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="kube-rbac-proxy" Oct 30 21:29:12.712305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.712102 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" containerName="init-config-reloader" Oct 30 21:29:12.742983 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.742964 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:29:12.743101 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.743089 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.747489 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.747372 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 21:29:12.747489 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.747412 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 21:29:12.747489 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.747448 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 21:29:12.747489 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.747471 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 21:29:12.751990 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.751966 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 21:29:12.751990 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.751985 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 21:29:12.752104 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.751980 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 21:29:12.752104 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.752031 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 21:29:12.752104 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.752040 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 21:29:12.752836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.752816 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 21:29:12.752836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.752836 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7mnf0orvh0m0j\"" Oct 30 21:29:12.752836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.752871 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 21:29:12.753141 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.752905 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 21:29:12.753141 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.753008 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 21:29:12.753141 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.753012 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 21:29:12.753141 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.753025 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vf75f\"" Oct 30 21:29:12.753141 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.753081 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 21:29:12.880086 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.880063 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a39962c3-e95b-4c79-bba1-f0c536c3575f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.880193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.880093 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a39962c3-e95b-4c79-bba1-f0c536c3575f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.880193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.880113 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.880193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.880127 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.880193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.880184 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a39962c3-e95b-4c79-bba1-f0c536c3575f-config-out\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.880363 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.880322 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a39962c3-e95b-4c79-bba1-f0c536c3575f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.880363 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.880352 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a39962c3-e95b-4c79-bba1-f0c536c3575f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.880451 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.880369 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.880451 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.880388 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.880451 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.880442 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a39962c3-e95b-4c79-bba1-f0c536c3575f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.880597 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.880496 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.880597 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.880519 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a39962c3-e95b-4c79-bba1-f0c536c3575f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.880597 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.880534 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-web-config\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.880597 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.880548 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a39962c3-e95b-4c79-bba1-f0c536c3575f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.880597 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.880572 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-config\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.880746 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.880650 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a39962c3-e95b-4c79-bba1-f0c536c3575f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.880746 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.880669 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a39962c3-e95b-4c79-bba1-f0c536c3575f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.880746 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.880714 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdd4f\" (UniqueName: \"kubernetes.io/projected/a39962c3-e95b-4c79-bba1-f0c536c3575f-kube-api-access-zdd4f\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.880868 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.880792 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.880868 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.880811 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.981767 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.981701 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a39962c3-e95b-4c79-bba1-f0c536c3575f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.981767 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.981730 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.981767 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.981745 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.981967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.981783 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a39962c3-e95b-4c79-bba1-f0c536c3575f-config-out\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.981967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.981805 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a39962c3-e95b-4c79-bba1-f0c536c3575f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.981967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.981829 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a39962c3-e95b-4c79-bba1-f0c536c3575f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.981967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.981846 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.981967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.981862 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.981967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.981883 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a39962c3-e95b-4c79-bba1-f0c536c3575f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.981967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.981912 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.981967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.981949 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a39962c3-e95b-4c79-bba1-f0c536c3575f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.982305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.981975 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-web-config\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.982305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.981996 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a39962c3-e95b-4c79-bba1-f0c536c3575f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.982305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.982030 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-config\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.982305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.982068 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a39962c3-e95b-4c79-bba1-f0c536c3575f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.982305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.982095 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a39962c3-e95b-4c79-bba1-f0c536c3575f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.982305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.982121 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdd4f\" (UniqueName: \"kubernetes.io/projected/a39962c3-e95b-4c79-bba1-f0c536c3575f-kube-api-access-zdd4f\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.982305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.982118 5801 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:29:12.982305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.982141 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.982305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.982180 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.982305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.982200 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a39962c3-e95b-4c79-bba1-f0c536c3575f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.982812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.982396 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a39962c3-e95b-4c79-bba1-f0c536c3575f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.982812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.982496 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a39962c3-e95b-4c79-bba1-f0c536c3575f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.982812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.982642 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a39962c3-e95b-4c79-bba1-f0c536c3575f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.982972 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.982958 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a39962c3-e95b-4c79-bba1-f0c536c3575f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.983481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.983455 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a39962c3-e95b-4c79-bba1-f0c536c3575f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.985033 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.985008 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a39962c3-e95b-4c79-bba1-f0c536c3575f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.985124 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.985073 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.985186 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.985167 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-web-config\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.985239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.985221 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.986215 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.985902 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a39962c3-e95b-4c79-bba1-f0c536c3575f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.986215 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.986018 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a39962c3-e95b-4c79-bba1-f0c536c3575f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.986215 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.986041 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.986431 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.986372 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-config\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.986825 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.986800 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.987038 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.987008 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a39962c3-e95b-4c79-bba1-f0c536c3575f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.987441 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.987419 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.987854 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.987835 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a39962c3-e95b-4c79-bba1-f0c536c3575f-config-out\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.987929 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.987849 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.988451 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.988437 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a39962c3-e95b-4c79-bba1-f0c536c3575f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:12.997294 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:12.997273 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdd4f\" (UniqueName: \"kubernetes.io/projected/a39962c3-e95b-4c79-bba1-f0c536c3575f-kube-api-access-zdd4f\") pod \"prometheus-k8s-0\" (UID: \"a39962c3-e95b-4c79-bba1-f0c536c3575f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:13.148614 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:13.148585 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:13.472902 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:13.472869 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:29:13.476230 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:29:13.476184 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39962c3_e95b_4c79_bba1_f0c536c3575f.slice/crio-47dccc4fdf6b3aeded5b6f6baae8dbbd1e78a441a6557c97944c8783b554ff00 WatchSource:0}: Error finding container 47dccc4fdf6b3aeded5b6f6baae8dbbd1e78a441a6557c97944c8783b554ff00: Status 404 returned error can't find the container with id 47dccc4fdf6b3aeded5b6f6baae8dbbd1e78a441a6557c97944c8783b554ff00 Oct 30 21:29:13.525592 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:13.525560 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="999c5bdd-e3fc-4103-9d65-0d3a228fc78b" path="/var/lib/kubelet/pods/999c5bdd-e3fc-4103-9d65-0d3a228fc78b/volumes" Oct 30 21:29:13.656728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:13.656701 5801 generic.go:356] "Generic (PLEG): container finished" podID="a39962c3-e95b-4c79-bba1-f0c536c3575f" containerID="8f72af6d87c4c9772f69a20cf8ef936622473c463347cd6b4a9cb6b64c2eab01" exitCode=0 Oct 30 21:29:13.657117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:13.656767 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a39962c3-e95b-4c79-bba1-f0c536c3575f","Type":"ContainerDied","Data":"8f72af6d87c4c9772f69a20cf8ef936622473c463347cd6b4a9cb6b64c2eab01"} Oct 30 21:29:13.657117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:13.656846 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a39962c3-e95b-4c79-bba1-f0c536c3575f","Type":"ContainerStarted","Data":"47dccc4fdf6b3aeded5b6f6baae8dbbd1e78a441a6557c97944c8783b554ff00"} Oct 30 21:29:14.663769 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:14.663717 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a39962c3-e95b-4c79-bba1-f0c536c3575f","Type":"ContainerStarted","Data":"aabe3314ff0eca979656974aa336f22f8df779ef19608f77ec6d31b16bf01b0c"} Oct 30 21:29:14.664223 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:14.663794 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a39962c3-e95b-4c79-bba1-f0c536c3575f","Type":"ContainerStarted","Data":"bd2b26a8297581b7f4b907b9c2ae844bb0ca53c2e901de114403b6889ebb338a"} Oct 30 21:29:14.664223 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:14.663820 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a39962c3-e95b-4c79-bba1-f0c536c3575f","Type":"ContainerStarted","Data":"a79206b1c046dd18f1295b0024a90d7886a2e0102a967d0b8726e2f5e59f989d"} Oct 30 21:29:14.664223 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:14.663846 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a39962c3-e95b-4c79-bba1-f0c536c3575f","Type":"ContainerStarted","Data":"e3859f44beeb426178ca576216d0a7265ffffeb7739ba66ae36be77e14933269"} Oct 30 21:29:14.664223 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:14.663870 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a39962c3-e95b-4c79-bba1-f0c536c3575f","Type":"ContainerStarted","Data":"a5d8377f81fdadbb926a6ed2794daceec22481ff9fa4e6906d0bc5e42602acd0"} Oct 30 21:29:14.664223 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:14.663891 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a39962c3-e95b-4c79-bba1-f0c536c3575f","Type":"ContainerStarted","Data":"f1ef3cc70ab080e9298a8db604289a561b10bcf8aa97336b41e02db14745194a"} Oct 30 21:29:14.709616 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:14.709555 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.6898184819999997 podStartE2EDuration="2.709540321s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.721839ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:14.708385282 +0000 UTC m=+23.304201941" watchObservedRunningTime="2026-10-30 21:29:14.709540321 +0000 UTC m=+23.305356975" Oct 30 21:29:14.819702 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:14.819669 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:14.923728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:14.923665 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:18.149274 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:18.149235 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:18.598945 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:18.598885 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-54ddb8d6b-4rlwf" podUID="189b46e5-b36a-4ce4-be3e-b5486b7e45a8" containerName="console" containerID="cri-o://d1335f87de32c5c790596fddc84b76e1f6055e69ce04d0c51156c382c0a9d20f" gracePeriod=15 Oct 30 21:29:18.834377 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:18.834352 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:29:18.921379 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:18.921320 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-service-ca\" (UniqueName: \"kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-service-ca\") pod \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " Oct 30 21:29:18.921379 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:18.921356 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-kube-api-access-sl65r\" (UniqueName: \"kubernetes.io/projected/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-kube-api-access-sl65r\") pod \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " Oct 30 21:29:18.921531 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:18.921387 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-oauth-config\") pod \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " Oct 30 21:29:18.921531 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:18.921500 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-tmp\" (UniqueName: \"kubernetes.io/empty-dir/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-tmp\") pod \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " Oct 30 21:29:18.921600 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:18.921547 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-serving-cert\") pod \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " Oct 30 21:29:18.921600 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:18.921584 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-config\" (UniqueName: \"kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-config\") pod \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " Oct 30 21:29:18.921700 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:18.921657 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-oauth-serving-cert\") pod \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\" (UID: \"189b46e5-b36a-4ce4-be3e-b5486b7e45a8\") " Oct 30 21:29:18.921964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:18.921744 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-service-ca" pod "189b46e5-b36a-4ce4-be3e-b5486b7e45a8" (UID: "189b46e5-b36a-4ce4-be3e-b5486b7e45a8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:18.921964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:18.921744 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-tmp" pod "189b46e5-b36a-4ce4-be3e-b5486b7e45a8" (UID: "189b46e5-b36a-4ce4-be3e-b5486b7e45a8"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:29:18.921964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:18.921949 5801 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:18.922172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:18.921969 5801 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-tmp\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:18.922172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:18.922036 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-config" pod "189b46e5-b36a-4ce4-be3e-b5486b7e45a8" (UID: "189b46e5-b36a-4ce4-be3e-b5486b7e45a8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:18.922172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:18.922078 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-oauth-serving-cert" pod "189b46e5-b36a-4ce4-be3e-b5486b7e45a8" (UID: "189b46e5-b36a-4ce4-be3e-b5486b7e45a8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:18.923789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:18.923744 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-serving-cert" pod "189b46e5-b36a-4ce4-be3e-b5486b7e45a8" (UID: "189b46e5-b36a-4ce4-be3e-b5486b7e45a8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:18.923888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:18.923795 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-kube-api-access-sl65r" pod "189b46e5-b36a-4ce4-be3e-b5486b7e45a8" (UID: "189b46e5-b36a-4ce4-be3e-b5486b7e45a8"). InnerVolumeSpecName "kube-api-access-sl65r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:29:18.923888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:18.923800 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-oauth-config" pod "189b46e5-b36a-4ce4-be3e-b5486b7e45a8" (UID: "189b46e5-b36a-4ce4-be3e-b5486b7e45a8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:19.022404 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:19.022376 5801 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-oauth-serving-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:19.022404 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:19.022401 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl65r\" (UniqueName: \"kubernetes.io/projected/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-kube-api-access-sl65r\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:19.022542 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:19.022418 5801 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-oauth-config\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:19.022542 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:19.022432 5801 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-serving-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:19.022542 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:19.022446 5801 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/189b46e5-b36a-4ce4-be3e-b5486b7e45a8-console-config\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:19.139321 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:19.139294 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c5fdfbf9b-l7clr"] Oct 30 21:29:19.682465 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:19.682433 5801 generic.go:356] "Generic (PLEG): container finished" podID="189b46e5-b36a-4ce4-be3e-b5486b7e45a8" containerID="d1335f87de32c5c790596fddc84b76e1f6055e69ce04d0c51156c382c0a9d20f" exitCode=0 Oct 30 21:29:19.682883 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:19.682471 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54ddb8d6b-4rlwf" Oct 30 21:29:19.682883 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:19.682488 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54ddb8d6b-4rlwf" event={"ID":"189b46e5-b36a-4ce4-be3e-b5486b7e45a8","Type":"ContainerDied","Data":"d1335f87de32c5c790596fddc84b76e1f6055e69ce04d0c51156c382c0a9d20f"} Oct 30 21:29:19.682883 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:19.682527 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54ddb8d6b-4rlwf" event={"ID":"189b46e5-b36a-4ce4-be3e-b5486b7e45a8","Type":"ContainerDied","Data":"a5510e59bc4d70b7949d7a6c736e5765e4da41eecb6bfbfed83b31498a7c5426"} Oct 30 21:29:19.682883 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:19.682552 5801 scope.go:122] "RemoveContainer" containerID="d1335f87de32c5c790596fddc84b76e1f6055e69ce04d0c51156c382c0a9d20f" Oct 30 21:29:19.692221 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:19.692196 5801 scope.go:122] "RemoveContainer" containerID="d1335f87de32c5c790596fddc84b76e1f6055e69ce04d0c51156c382c0a9d20f" Oct 30 21:29:19.693718 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:29:19.693678 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1335f87de32c5c790596fddc84b76e1f6055e69ce04d0c51156c382c0a9d20f\": container with ID starting with d1335f87de32c5c790596fddc84b76e1f6055e69ce04d0c51156c382c0a9d20f not found: ID does not exist" containerID="d1335f87de32c5c790596fddc84b76e1f6055e69ce04d0c51156c382c0a9d20f" Oct 30 21:29:19.693812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:19.693721 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1335f87de32c5c790596fddc84b76e1f6055e69ce04d0c51156c382c0a9d20f"} err="failed to get container status \"d1335f87de32c5c790596fddc84b76e1f6055e69ce04d0c51156c382c0a9d20f\": rpc error: code = NotFound desc = could not find container \"d1335f87de32c5c790596fddc84b76e1f6055e69ce04d0c51156c382c0a9d20f\": container with ID starting with d1335f87de32c5c790596fddc84b76e1f6055e69ce04d0c51156c382c0a9d20f not found: ID does not exist" Oct 30 21:29:19.702728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:19.702708 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54ddb8d6b-4rlwf"] Oct 30 21:29:19.711674 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:19.711654 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54ddb8d6b-4rlwf"] Oct 30 21:29:21.521470 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:21.521424 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="189b46e5-b36a-4ce4-be3e-b5486b7e45a8" path="/var/lib/kubelet/pods/189b46e5-b36a-4ce4-be3e-b5486b7e45a8/volumes" Oct 30 21:29:25.676609 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:25.676575 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:25.692633 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:25.692606 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:25.715734 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:25.715710 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:29:27.569891 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:27.569862 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kv4cs" Oct 30 21:29:30.275939 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:29:30.275898 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:29:44.157925 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.157859 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5c5fdfbf9b-l7clr" podUID="8d43f954-119d-40c5-9727-1e25043da4ee" containerName="console" containerID="cri-o://d30ed4895ec7878e057cb57171fae40237e2a25e793c73814789d39ed11a5de7" gracePeriod=15 Oct 30 21:29:44.392623 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.392599 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:29:44.405829 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.405809 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d43f954-119d-40c5-9727-1e25043da4ee-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d43f954-119d-40c5-9727-1e25043da4ee-tmp\") pod \"8d43f954-119d-40c5-9727-1e25043da4ee\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " Oct 30 21:29:44.405930 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.405863 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-trusted-ca-bundle\") pod \"8d43f954-119d-40c5-9727-1e25043da4ee\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " Oct 30 21:29:44.405930 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.405920 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d43f954-119d-40c5-9727-1e25043da4ee-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d43f954-119d-40c5-9727-1e25043da4ee-console-serving-cert\") pod \"8d43f954-119d-40c5-9727-1e25043da4ee\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " Oct 30 21:29:44.406015 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.405944 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d43f954-119d-40c5-9727-1e25043da4ee-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d43f954-119d-40c5-9727-1e25043da4ee-console-oauth-config\") pod \"8d43f954-119d-40c5-9727-1e25043da4ee\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " Oct 30 21:29:44.406083 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.406057 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d43f954-119d-40c5-9727-1e25043da4ee-tmp" pod "8d43f954-119d-40c5-9727-1e25043da4ee" (UID: "8d43f954-119d-40c5-9727-1e25043da4ee"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:29:44.406143 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.406083 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-console-config\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-console-config\") pod \"8d43f954-119d-40c5-9727-1e25043da4ee\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " Oct 30 21:29:44.406143 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.406127 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-service-ca\") pod \"8d43f954-119d-40c5-9727-1e25043da4ee\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " Oct 30 21:29:44.406251 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.406172 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-oauth-serving-cert\") pod \"8d43f954-119d-40c5-9727-1e25043da4ee\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " Oct 30 21:29:44.406308 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.406285 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d43f954-119d-40c5-9727-1e25043da4ee-kube-api-access-bczbc\" (UniqueName: \"kubernetes.io/projected/8d43f954-119d-40c5-9727-1e25043da4ee-kube-api-access-bczbc\") pod \"8d43f954-119d-40c5-9727-1e25043da4ee\" (UID: \"8d43f954-119d-40c5-9727-1e25043da4ee\") " Oct 30 21:29:44.406538 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.406518 5801 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d43f954-119d-40c5-9727-1e25043da4ee-tmp\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:44.406538 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.406286 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-trusted-ca-bundle" pod "8d43f954-119d-40c5-9727-1e25043da4ee" (UID: "8d43f954-119d-40c5-9727-1e25043da4ee"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:44.406647 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.406449 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-console-config" pod "8d43f954-119d-40c5-9727-1e25043da4ee" (UID: "8d43f954-119d-40c5-9727-1e25043da4ee"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:44.406647 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.406459 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-service-ca" pod "8d43f954-119d-40c5-9727-1e25043da4ee" (UID: "8d43f954-119d-40c5-9727-1e25043da4ee"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:44.406647 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.406490 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-oauth-serving-cert" pod "8d43f954-119d-40c5-9727-1e25043da4ee" (UID: "8d43f954-119d-40c5-9727-1e25043da4ee"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:44.408062 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.408006 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d43f954-119d-40c5-9727-1e25043da4ee-console-oauth-config" pod "8d43f954-119d-40c5-9727-1e25043da4ee" (UID: "8d43f954-119d-40c5-9727-1e25043da4ee"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:44.408154 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.408077 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d43f954-119d-40c5-9727-1e25043da4ee-console-serving-cert" pod "8d43f954-119d-40c5-9727-1e25043da4ee" (UID: "8d43f954-119d-40c5-9727-1e25043da4ee"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:44.408208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.408190 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d43f954-119d-40c5-9727-1e25043da4ee-kube-api-access-bczbc" pod "8d43f954-119d-40c5-9727-1e25043da4ee" (UID: "8d43f954-119d-40c5-9727-1e25043da4ee"). InnerVolumeSpecName "kube-api-access-bczbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:29:44.507369 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.507346 5801 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d43f954-119d-40c5-9727-1e25043da4ee-console-oauth-config\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:44.507369 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.507368 5801 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-console-config\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:44.507500 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.507377 5801 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:44.507500 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.507387 5801 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-oauth-serving-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:44.507500 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.507396 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bczbc\" (UniqueName: \"kubernetes.io/projected/8d43f954-119d-40c5-9727-1e25043da4ee-kube-api-access-bczbc\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:44.507500 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.507406 5801 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d43f954-119d-40c5-9727-1e25043da4ee-trusted-ca-bundle\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:44.507500 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.507415 5801 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d43f954-119d-40c5-9727-1e25043da4ee-console-serving-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:44.762030 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.761960 5801 generic.go:356] "Generic (PLEG): container finished" podID="8d43f954-119d-40c5-9727-1e25043da4ee" containerID="d30ed4895ec7878e057cb57171fae40237e2a25e793c73814789d39ed11a5de7" exitCode=0 Oct 30 21:29:44.762030 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.762004 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5fdfbf9b-l7clr" Oct 30 21:29:44.762030 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.762015 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5fdfbf9b-l7clr" event={"ID":"8d43f954-119d-40c5-9727-1e25043da4ee","Type":"ContainerDied","Data":"d30ed4895ec7878e057cb57171fae40237e2a25e793c73814789d39ed11a5de7"} Oct 30 21:29:44.762241 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.762054 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5fdfbf9b-l7clr" event={"ID":"8d43f954-119d-40c5-9727-1e25043da4ee","Type":"ContainerDied","Data":"bcd8b205dfbf9b7c1e054114904f54e7451342bcfa11344e056753f200b7694b"} Oct 30 21:29:44.762241 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.762074 5801 scope.go:122] "RemoveContainer" containerID="d30ed4895ec7878e057cb57171fae40237e2a25e793c73814789d39ed11a5de7" Oct 30 21:29:44.771973 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.771952 5801 scope.go:122] "RemoveContainer" containerID="d30ed4895ec7878e057cb57171fae40237e2a25e793c73814789d39ed11a5de7" Oct 30 21:29:44.773488 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:29:44.773466 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d30ed4895ec7878e057cb57171fae40237e2a25e793c73814789d39ed11a5de7\": container with ID starting with d30ed4895ec7878e057cb57171fae40237e2a25e793c73814789d39ed11a5de7 not found: ID does not exist" containerID="d30ed4895ec7878e057cb57171fae40237e2a25e793c73814789d39ed11a5de7" Oct 30 21:29:44.773569 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.773496 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d30ed4895ec7878e057cb57171fae40237e2a25e793c73814789d39ed11a5de7"} err="failed to get container status \"d30ed4895ec7878e057cb57171fae40237e2a25e793c73814789d39ed11a5de7\": rpc error: code = NotFound desc = could not find container \"d30ed4895ec7878e057cb57171fae40237e2a25e793c73814789d39ed11a5de7\": container with ID starting with d30ed4895ec7878e057cb57171fae40237e2a25e793c73814789d39ed11a5de7 not found: ID does not exist" Oct 30 21:29:44.786201 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.786180 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c5fdfbf9b-l7clr"] Oct 30 21:29:44.790800 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:44.790778 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c5fdfbf9b-l7clr"] Oct 30 21:29:45.525194 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:45.525156 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d43f954-119d-40c5-9727-1e25043da4ee" path="/var/lib/kubelet/pods/8d43f954-119d-40c5-9727-1e25043da4ee/volumes" Oct 30 21:29:54.298955 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:54.298915 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:54.401208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:54.401171 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:56.523891 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:29:56.523852 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:00.277273 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:30:00.277233 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:30:12.892836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:30:12.892742 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:12.990436 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:30:12.990395 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:30.278071 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:30:30.278032 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:30:56.583028 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:30:56.582943 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:58.844200 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:30:58.844165 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:58.984093 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:30:58.984059 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:59.113944 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:30:59.113845 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:00.277433 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:31:00.277397 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:31:00.407796 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:00.407738 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:00.509730 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:00.509690 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:01.757452 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:01.757408 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:02.842088 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:02.842056 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml"] Oct 30 21:31:02.842500 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:02.842327 5801 cpu_manager.go:414] "containerMap: removing container" podUID="8d43f954-119d-40c5-9727-1e25043da4ee" containerName="console" Oct 30 21:31:02.842500 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:02.842338 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d43f954-119d-40c5-9727-1e25043da4ee" containerName="console" Oct 30 21:31:02.842500 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:02.842350 5801 cpu_manager.go:414] "containerMap: removing container" podUID="189b46e5-b36a-4ce4-be3e-b5486b7e45a8" containerName="console" Oct 30 21:31:02.842500 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:02.842355 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189b46e5-b36a-4ce4-be3e-b5486b7e45a8" containerName="console" Oct 30 21:31:02.842500 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:02.842410 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d43f954-119d-40c5-9727-1e25043da4ee" containerName="console" Oct 30 21:31:02.842500 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:02.842418 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="189b46e5-b36a-4ce4-be3e-b5486b7e45a8" containerName="console" Oct 30 21:31:02.845273 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:02.845251 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml" Oct 30 21:31:02.850237 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:02.850201 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-kq5hz\"" Oct 30 21:31:02.850391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:02.850209 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 21:31:02.850391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:02.850232 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 21:31:02.854906 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:02.854884 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml"] Oct 30 21:31:02.868768 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:02.868732 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4krsr\" (UniqueName: \"kubernetes.io/projected/1b9a5b39-38d0-4ef1-aa12-4d0453fdb039-kube-api-access-4krsr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml\" (UID: \"1b9a5b39-38d0-4ef1-aa12-4d0453fdb039\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml" Oct 30 21:31:02.868881 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:02.868819 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b9a5b39-38d0-4ef1-aa12-4d0453fdb039-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml\" (UID: \"1b9a5b39-38d0-4ef1-aa12-4d0453fdb039\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml" Oct 30 21:31:02.868881 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:02.868872 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b9a5b39-38d0-4ef1-aa12-4d0453fdb039-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml\" (UID: \"1b9a5b39-38d0-4ef1-aa12-4d0453fdb039\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml" Oct 30 21:31:02.969822 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:02.969783 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4krsr\" (UniqueName: \"kubernetes.io/projected/1b9a5b39-38d0-4ef1-aa12-4d0453fdb039-kube-api-access-4krsr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml\" (UID: \"1b9a5b39-38d0-4ef1-aa12-4d0453fdb039\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml" Oct 30 21:31:02.969822 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:02.969828 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b9a5b39-38d0-4ef1-aa12-4d0453fdb039-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml\" (UID: \"1b9a5b39-38d0-4ef1-aa12-4d0453fdb039\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml" Oct 30 21:31:02.970078 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:02.969869 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b9a5b39-38d0-4ef1-aa12-4d0453fdb039-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml\" (UID: \"1b9a5b39-38d0-4ef1-aa12-4d0453fdb039\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml" Oct 30 21:31:02.970189 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:02.970167 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b9a5b39-38d0-4ef1-aa12-4d0453fdb039-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml\" (UID: \"1b9a5b39-38d0-4ef1-aa12-4d0453fdb039\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml" Oct 30 21:31:02.970241 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:02.970193 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b9a5b39-38d0-4ef1-aa12-4d0453fdb039-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml\" (UID: \"1b9a5b39-38d0-4ef1-aa12-4d0453fdb039\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml" Oct 30 21:31:02.984478 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:02.984450 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4krsr\" (UniqueName: \"kubernetes.io/projected/1b9a5b39-38d0-4ef1-aa12-4d0453fdb039-kube-api-access-4krsr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml\" (UID: \"1b9a5b39-38d0-4ef1-aa12-4d0453fdb039\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml" Oct 30 21:31:03.160743 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.160665 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml" Oct 30 21:31:03.431178 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.431107 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml"] Oct 30 21:31:03.434513 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:31:03.434485 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9a5b39_38d0_4ef1_aa12_4d0453fdb039.slice/crio-d151f1d56874b5f79d3bdd3eed73d0fca080e6ef96f6d3f3e3307ca80f9f2b44 WatchSource:0}: Error finding container d151f1d56874b5f79d3bdd3eed73d0fca080e6ef96f6d3f3e3307ca80f9f2b44: Status 404 returned error can't find the container with id d151f1d56874b5f79d3bdd3eed73d0fca080e6ef96f6d3f3e3307ca80f9f2b44 Oct 30 21:31:03.436590 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.436574 5801 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:31:03.523538 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.523509 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw"] Oct 30 21:31:03.527914 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.527895 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw" Oct 30 21:31:03.531244 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.531220 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-c75r9\"" Oct 30 21:31:03.537088 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.537064 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw"] Oct 30 21:31:03.573625 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.573594 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmlc2\" (UniqueName: \"kubernetes.io/projected/67973dda-956d-4c9e-895f-02862eb2dc08-kube-api-access-cmlc2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw\" (UID: \"67973dda-956d-4c9e-895f-02862eb2dc08\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw" Oct 30 21:31:03.573796 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.573628 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67973dda-956d-4c9e-895f-02862eb2dc08-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw\" (UID: \"67973dda-956d-4c9e-895f-02862eb2dc08\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw" Oct 30 21:31:03.573796 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.573651 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67973dda-956d-4c9e-895f-02862eb2dc08-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw\" (UID: \"67973dda-956d-4c9e-895f-02862eb2dc08\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw" Oct 30 21:31:03.623083 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.623001 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn"] Oct 30 21:31:03.626293 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.626277 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn" Oct 30 21:31:03.629688 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.629664 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-pxcwj\"" Oct 30 21:31:03.636425 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.636403 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn"] Oct 30 21:31:03.674222 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.674187 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zjq2\" (UniqueName: \"kubernetes.io/projected/685bdc58-e232-4570-89ca-388bed110bc1-kube-api-access-8zjq2\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn\" (UID: \"685bdc58-e232-4570-89ca-388bed110bc1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn" Oct 30 21:31:03.674363 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.674228 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/685bdc58-e232-4570-89ca-388bed110bc1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn\" (UID: \"685bdc58-e232-4570-89ca-388bed110bc1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn" Oct 30 21:31:03.674363 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.674309 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmlc2\" (UniqueName: \"kubernetes.io/projected/67973dda-956d-4c9e-895f-02862eb2dc08-kube-api-access-cmlc2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw\" (UID: \"67973dda-956d-4c9e-895f-02862eb2dc08\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw" Oct 30 21:31:03.674363 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.674345 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67973dda-956d-4c9e-895f-02862eb2dc08-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw\" (UID: \"67973dda-956d-4c9e-895f-02862eb2dc08\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw" Oct 30 21:31:03.674466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.674366 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67973dda-956d-4c9e-895f-02862eb2dc08-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw\" (UID: \"67973dda-956d-4c9e-895f-02862eb2dc08\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw" Oct 30 21:31:03.674466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.674396 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/685bdc58-e232-4570-89ca-388bed110bc1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn\" (UID: \"685bdc58-e232-4570-89ca-388bed110bc1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn" Oct 30 21:31:03.674685 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.674671 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67973dda-956d-4c9e-895f-02862eb2dc08-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw\" (UID: \"67973dda-956d-4c9e-895f-02862eb2dc08\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw" Oct 30 21:31:03.674722 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.674692 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67973dda-956d-4c9e-895f-02862eb2dc08-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw\" (UID: \"67973dda-956d-4c9e-895f-02862eb2dc08\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw" Oct 30 21:31:03.686642 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.686574 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmlc2\" (UniqueName: \"kubernetes.io/projected/67973dda-956d-4c9e-895f-02862eb2dc08-kube-api-access-cmlc2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw\" (UID: \"67973dda-956d-4c9e-895f-02862eb2dc08\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw" Oct 30 21:31:03.722822 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.722792 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5"] Oct 30 21:31:03.726202 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.726181 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5" Oct 30 21:31:03.729845 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.729822 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-5qx7c\"" Oct 30 21:31:03.736006 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.735983 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5"] Oct 30 21:31:03.775318 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.775284 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gk5r\" (UniqueName: \"kubernetes.io/projected/20c970ef-b259-4d62-9f7c-3d80a2ef7405-kube-api-access-6gk5r\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5\" (UID: \"20c970ef-b259-4d62-9f7c-3d80a2ef7405\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5" Oct 30 21:31:03.775318 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.775322 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zjq2\" (UniqueName: \"kubernetes.io/projected/685bdc58-e232-4570-89ca-388bed110bc1-kube-api-access-8zjq2\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn\" (UID: \"685bdc58-e232-4570-89ca-388bed110bc1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn" Oct 30 21:31:03.775556 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.775348 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/685bdc58-e232-4570-89ca-388bed110bc1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn\" (UID: \"685bdc58-e232-4570-89ca-388bed110bc1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn" Oct 30 21:31:03.775556 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.775415 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20c970ef-b259-4d62-9f7c-3d80a2ef7405-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5\" (UID: \"20c970ef-b259-4d62-9f7c-3d80a2ef7405\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5" Oct 30 21:31:03.775556 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.775450 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20c970ef-b259-4d62-9f7c-3d80a2ef7405-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5\" (UID: \"20c970ef-b259-4d62-9f7c-3d80a2ef7405\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5" Oct 30 21:31:03.775713 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.775605 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/685bdc58-e232-4570-89ca-388bed110bc1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn\" (UID: \"685bdc58-e232-4570-89ca-388bed110bc1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn" Oct 30 21:31:03.775881 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.775863 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/685bdc58-e232-4570-89ca-388bed110bc1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn\" (UID: \"685bdc58-e232-4570-89ca-388bed110bc1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn" Oct 30 21:31:03.775965 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.775943 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/685bdc58-e232-4570-89ca-388bed110bc1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn\" (UID: \"685bdc58-e232-4570-89ca-388bed110bc1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn" Oct 30 21:31:03.788008 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.787987 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zjq2\" (UniqueName: \"kubernetes.io/projected/685bdc58-e232-4570-89ca-388bed110bc1-kube-api-access-8zjq2\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn\" (UID: \"685bdc58-e232-4570-89ca-388bed110bc1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn" Oct 30 21:31:03.823837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.823804 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x"] Oct 30 21:31:03.827047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.827031 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x" Oct 30 21:31:03.830834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.830812 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-nllpv\"" Oct 30 21:31:03.837286 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.837264 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x"] Oct 30 21:31:03.844516 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.844495 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw" Oct 30 21:31:03.876688 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.876655 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1fc8b20-9a67-4298-aa95-f07368cafb1f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x\" (UID: \"b1fc8b20-9a67-4298-aa95-f07368cafb1f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x" Oct 30 21:31:03.876881 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.876699 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1fc8b20-9a67-4298-aa95-f07368cafb1f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x\" (UID: \"b1fc8b20-9a67-4298-aa95-f07368cafb1f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x" Oct 30 21:31:03.876881 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.876725 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gk5r\" (UniqueName: \"kubernetes.io/projected/20c970ef-b259-4d62-9f7c-3d80a2ef7405-kube-api-access-6gk5r\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5\" (UID: \"20c970ef-b259-4d62-9f7c-3d80a2ef7405\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5" Oct 30 21:31:03.877012 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.876897 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2xf5\" (UniqueName: \"kubernetes.io/projected/b1fc8b20-9a67-4298-aa95-f07368cafb1f-kube-api-access-l2xf5\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x\" (UID: \"b1fc8b20-9a67-4298-aa95-f07368cafb1f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x" Oct 30 21:31:03.877012 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.876957 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20c970ef-b259-4d62-9f7c-3d80a2ef7405-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5\" (UID: \"20c970ef-b259-4d62-9f7c-3d80a2ef7405\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5" Oct 30 21:31:03.877012 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.876998 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20c970ef-b259-4d62-9f7c-3d80a2ef7405-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5\" (UID: \"20c970ef-b259-4d62-9f7c-3d80a2ef7405\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5" Oct 30 21:31:03.877384 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.877364 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20c970ef-b259-4d62-9f7c-3d80a2ef7405-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5\" (UID: \"20c970ef-b259-4d62-9f7c-3d80a2ef7405\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5" Oct 30 21:31:03.877461 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.877379 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20c970ef-b259-4d62-9f7c-3d80a2ef7405-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5\" (UID: \"20c970ef-b259-4d62-9f7c-3d80a2ef7405\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5" Oct 30 21:31:03.889698 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.889669 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gk5r\" (UniqueName: \"kubernetes.io/projected/20c970ef-b259-4d62-9f7c-3d80a2ef7405-kube-api-access-6gk5r\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5\" (UID: \"20c970ef-b259-4d62-9f7c-3d80a2ef7405\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5" Oct 30 21:31:03.941597 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.941519 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn" Oct 30 21:31:03.977732 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.977691 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1fc8b20-9a67-4298-aa95-f07368cafb1f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x\" (UID: \"b1fc8b20-9a67-4298-aa95-f07368cafb1f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x" Oct 30 21:31:03.977885 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.977770 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2xf5\" (UniqueName: \"kubernetes.io/projected/b1fc8b20-9a67-4298-aa95-f07368cafb1f-kube-api-access-l2xf5\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x\" (UID: \"b1fc8b20-9a67-4298-aa95-f07368cafb1f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x" Oct 30 21:31:03.977885 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.977839 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1fc8b20-9a67-4298-aa95-f07368cafb1f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x\" (UID: \"b1fc8b20-9a67-4298-aa95-f07368cafb1f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x" Oct 30 21:31:03.978225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.978206 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1fc8b20-9a67-4298-aa95-f07368cafb1f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x\" (UID: \"b1fc8b20-9a67-4298-aa95-f07368cafb1f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x" Oct 30 21:31:03.978462 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.978438 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1fc8b20-9a67-4298-aa95-f07368cafb1f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x\" (UID: \"b1fc8b20-9a67-4298-aa95-f07368cafb1f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x" Oct 30 21:31:03.991653 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:03.991629 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2xf5\" (UniqueName: \"kubernetes.io/projected/b1fc8b20-9a67-4298-aa95-f07368cafb1f-kube-api-access-l2xf5\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x\" (UID: \"b1fc8b20-9a67-4298-aa95-f07368cafb1f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x" Oct 30 21:31:04.017276 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:04.017245 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml" event={"ID":"1b9a5b39-38d0-4ef1-aa12-4d0453fdb039","Type":"ContainerStarted","Data":"d151f1d56874b5f79d3bdd3eed73d0fca080e6ef96f6d3f3e3307ca80f9f2b44"} Oct 30 21:31:04.042420 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:04.042389 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5" Oct 30 21:31:04.127290 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:04.127251 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw"] Oct 30 21:31:04.130734 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:31:04.130700 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67973dda_956d_4c9e_895f_02862eb2dc08.slice/crio-fdbc47d7c67a500ef1dabdcc7aa26bb0e9533cd23ee15a50516586fd48e625d9 WatchSource:0}: Error finding container fdbc47d7c67a500ef1dabdcc7aa26bb0e9533cd23ee15a50516586fd48e625d9: Status 404 returned error can't find the container with id fdbc47d7c67a500ef1dabdcc7aa26bb0e9533cd23ee15a50516586fd48e625d9 Oct 30 21:31:04.142883 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:04.142859 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x" Oct 30 21:31:04.224779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:04.224737 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn"] Oct 30 21:31:04.231688 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:31:04.231662 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod685bdc58_e232_4570_89ca_388bed110bc1.slice/crio-e2bdfceedbc58d1ac4852c13f1ea45cef8ee5bb428a9eafabb14ebabbe67269d WatchSource:0}: Error finding container e2bdfceedbc58d1ac4852c13f1ea45cef8ee5bb428a9eafabb14ebabbe67269d: Status 404 returned error can't find the container with id e2bdfceedbc58d1ac4852c13f1ea45cef8ee5bb428a9eafabb14ebabbe67269d Oct 30 21:31:04.326394 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:04.326302 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5"] Oct 30 21:31:04.330521 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:31:04.330489 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c970ef_b259_4d62_9f7c_3d80a2ef7405.slice/crio-33b7870c91f30c877a3fb74c06b7a305cfc8955ba7cae21cfbd30c0b5c557756 WatchSource:0}: Error finding container 33b7870c91f30c877a3fb74c06b7a305cfc8955ba7cae21cfbd30c0b5c557756: Status 404 returned error can't find the container with id 33b7870c91f30c877a3fb74c06b7a305cfc8955ba7cae21cfbd30c0b5c557756 Oct 30 21:31:04.422372 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:04.422341 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x"] Oct 30 21:31:04.427176 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:31:04.427148 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1fc8b20_9a67_4298_aa95_f07368cafb1f.slice/crio-c914a0c9392f537ec9aab97814fedb3aa3e545789abdbd2d17a9d5946e71d207 WatchSource:0}: Error finding container c914a0c9392f537ec9aab97814fedb3aa3e545789abdbd2d17a9d5946e71d207: Status 404 returned error can't find the container with id c914a0c9392f537ec9aab97814fedb3aa3e545789abdbd2d17a9d5946e71d207 Oct 30 21:31:05.025895 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.025742 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x" event={"ID":"b1fc8b20-9a67-4298-aa95-f07368cafb1f","Type":"ContainerStarted","Data":"c914a0c9392f537ec9aab97814fedb3aa3e545789abdbd2d17a9d5946e71d207"} Oct 30 21:31:05.027779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.027689 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5" event={"ID":"20c970ef-b259-4d62-9f7c-3d80a2ef7405","Type":"ContainerStarted","Data":"33b7870c91f30c877a3fb74c06b7a305cfc8955ba7cae21cfbd30c0b5c557756"} Oct 30 21:31:05.030294 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.030233 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw" event={"ID":"67973dda-956d-4c9e-895f-02862eb2dc08","Type":"ContainerStarted","Data":"fdbc47d7c67a500ef1dabdcc7aa26bb0e9533cd23ee15a50516586fd48e625d9"} Oct 30 21:31:05.033238 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.033187 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn" event={"ID":"685bdc58-e232-4570-89ca-388bed110bc1","Type":"ContainerStarted","Data":"e2bdfceedbc58d1ac4852c13f1ea45cef8ee5bb428a9eafabb14ebabbe67269d"} Oct 30 21:31:05.307101 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.307000 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:05.414235 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.414142 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:05.660843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.659064 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd"] Oct 30 21:31:05.663869 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.663280 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd" Oct 30 21:31:05.676717 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.676025 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-787g4\"" Oct 30 21:31:05.678121 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.678097 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd"] Oct 30 21:31:05.694614 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.694578 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0781ba05-2ba2-4717-ba74-ef1244d51051-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd\" (UID: \"0781ba05-2ba2-4717-ba74-ef1244d51051\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd" Oct 30 21:31:05.694794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.694646 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0781ba05-2ba2-4717-ba74-ef1244d51051-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd\" (UID: \"0781ba05-2ba2-4717-ba74-ef1244d51051\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd" Oct 30 21:31:05.694794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.694679 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8mdr\" (UniqueName: \"kubernetes.io/projected/0781ba05-2ba2-4717-ba74-ef1244d51051-kube-api-access-c8mdr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd\" (UID: \"0781ba05-2ba2-4717-ba74-ef1244d51051\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd" Oct 30 21:31:05.795278 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.795240 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0781ba05-2ba2-4717-ba74-ef1244d51051-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd\" (UID: \"0781ba05-2ba2-4717-ba74-ef1244d51051\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd" Oct 30 21:31:05.795453 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.795293 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8mdr\" (UniqueName: \"kubernetes.io/projected/0781ba05-2ba2-4717-ba74-ef1244d51051-kube-api-access-c8mdr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd\" (UID: \"0781ba05-2ba2-4717-ba74-ef1244d51051\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd" Oct 30 21:31:05.795453 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.795362 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0781ba05-2ba2-4717-ba74-ef1244d51051-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd\" (UID: \"0781ba05-2ba2-4717-ba74-ef1244d51051\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd" Oct 30 21:31:05.795846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.795820 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0781ba05-2ba2-4717-ba74-ef1244d51051-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd\" (UID: \"0781ba05-2ba2-4717-ba74-ef1244d51051\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd" Oct 30 21:31:05.796098 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.796075 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0781ba05-2ba2-4717-ba74-ef1244d51051-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd\" (UID: \"0781ba05-2ba2-4717-ba74-ef1244d51051\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd" Oct 30 21:31:05.832530 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.832498 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8mdr\" (UniqueName: \"kubernetes.io/projected/0781ba05-2ba2-4717-ba74-ef1244d51051-kube-api-access-c8mdr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd\" (UID: \"0781ba05-2ba2-4717-ba74-ef1244d51051\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd" Oct 30 21:31:05.940221 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.938636 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg"] Oct 30 21:31:05.942660 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.942633 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg" Oct 30 21:31:05.948501 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.948462 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-wxl9z\"" Oct 30 21:31:05.954396 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.954337 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg"] Oct 30 21:31:05.986009 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.985979 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd" Oct 30 21:31:05.997553 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.997463 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39b20d2f-7f91-4a12-b2a6-8045b2879a0a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg\" (UID: \"39b20d2f-7f91-4a12-b2a6-8045b2879a0a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg" Oct 30 21:31:05.997553 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.997514 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39b20d2f-7f91-4a12-b2a6-8045b2879a0a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg\" (UID: \"39b20d2f-7f91-4a12-b2a6-8045b2879a0a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg" Oct 30 21:31:05.997733 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:05.997665 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2zqc\" (UniqueName: \"kubernetes.io/projected/39b20d2f-7f91-4a12-b2a6-8045b2879a0a-kube-api-access-r2zqc\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg\" (UID: \"39b20d2f-7f91-4a12-b2a6-8045b2879a0a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg" Oct 30 21:31:06.098118 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.098074 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2zqc\" (UniqueName: \"kubernetes.io/projected/39b20d2f-7f91-4a12-b2a6-8045b2879a0a-kube-api-access-r2zqc\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg\" (UID: \"39b20d2f-7f91-4a12-b2a6-8045b2879a0a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg" Oct 30 21:31:06.098577 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.098148 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39b20d2f-7f91-4a12-b2a6-8045b2879a0a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg\" (UID: \"39b20d2f-7f91-4a12-b2a6-8045b2879a0a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg" Oct 30 21:31:06.098577 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.098181 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39b20d2f-7f91-4a12-b2a6-8045b2879a0a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg\" (UID: \"39b20d2f-7f91-4a12-b2a6-8045b2879a0a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg" Oct 30 21:31:06.098701 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.098633 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39b20d2f-7f91-4a12-b2a6-8045b2879a0a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg\" (UID: \"39b20d2f-7f91-4a12-b2a6-8045b2879a0a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg" Oct 30 21:31:06.099164 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.099142 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39b20d2f-7f91-4a12-b2a6-8045b2879a0a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg\" (UID: \"39b20d2f-7f91-4a12-b2a6-8045b2879a0a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg" Oct 30 21:31:06.122094 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.121991 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2zqc\" (UniqueName: \"kubernetes.io/projected/39b20d2f-7f91-4a12-b2a6-8045b2879a0a-kube-api-access-r2zqc\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg\" (UID: \"39b20d2f-7f91-4a12-b2a6-8045b2879a0a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg" Oct 30 21:31:06.265013 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.264613 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg" Oct 30 21:31:06.279393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.279370 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd"] Oct 30 21:31:06.436777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.435290 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk"] Oct 30 21:31:06.442403 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.441169 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.446683 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.446625 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wb984\"" Oct 30 21:31:06.451951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.450740 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:31:06.451951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.450827 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 21:31:06.451951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.450909 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:31:06.451951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.451861 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk"] Oct 30 21:31:06.478201 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.478169 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:06.503405 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.503372 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.503574 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.503429 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d7a2c8-0585-48a3-88d4-d49b23672216-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.503574 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.503498 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.503574 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.503544 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.503787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.503601 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.503787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.503633 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.503787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.503682 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glp8f\" (UniqueName: \"kubernetes.io/projected/79d7a2c8-0585-48a3-88d4-d49b23672216-kube-api-access-glp8f\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.503787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.503717 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.503787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.503768 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.581677 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.581645 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg"] Oct 30 21:31:06.587791 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:31:06.587747 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b20d2f_7f91_4a12_b2a6_8045b2879a0a.slice/crio-bd9eddf466486bf5cfaf34061a6e97488a6a555e417beb1526163b0d1fb6ab4f WatchSource:0}: Error finding container bd9eddf466486bf5cfaf34061a6e97488a6a555e417beb1526163b0d1fb6ab4f: Status 404 returned error can't find the container with id bd9eddf466486bf5cfaf34061a6e97488a6a555e417beb1526163b0d1fb6ab4f Oct 30 21:31:06.589944 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.589915 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:06.604942 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.604903 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.605078 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.604975 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d7a2c8-0585-48a3-88d4-d49b23672216-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.605078 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.605007 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.605078 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.605053 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.605239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.605089 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.605239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.605120 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.605239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.605167 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glp8f\" (UniqueName: \"kubernetes.io/projected/79d7a2c8-0585-48a3-88d4-d49b23672216-kube-api-access-glp8f\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.605239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.605198 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.605239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.605227 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.605931 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.605905 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.606575 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.606543 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d7a2c8-0585-48a3-88d4-d49b23672216-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.606842 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.606817 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.608279 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.607455 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.608279 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.608243 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.611217 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.611126 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.617011 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.616986 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.629333 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.629296 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.631640 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.631618 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glp8f\" (UniqueName: \"kubernetes.io/projected/79d7a2c8-0585-48a3-88d4-d49b23672216-kube-api-access-glp8f\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-448hk\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:06.689462 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.689422 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:06.762096 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:06.762055 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:07.052252 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:07.052155 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg" event={"ID":"39b20d2f-7f91-4a12-b2a6-8045b2879a0a","Type":"ContainerStarted","Data":"bd9eddf466486bf5cfaf34061a6e97488a6a555e417beb1526163b0d1fb6ab4f"} Oct 30 21:31:07.054129 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:07.054057 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd" event={"ID":"0781ba05-2ba2-4717-ba74-ef1244d51051","Type":"ContainerStarted","Data":"d87f292196aae977f07b676bc4f39c1051702578685b9c7c9e4dd155ac7f370d"} Oct 30 21:31:07.083160 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:07.083069 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk"] Oct 30 21:31:08.066729 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:08.066689 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" event={"ID":"79d7a2c8-0585-48a3-88d4-d49b23672216","Type":"ContainerStarted","Data":"211580f9c8bdab1d35270f9f520a86742462f21574f6f2039066261a78b93f62"} Oct 30 21:31:09.411924 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:09.411883 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:15.515925 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:15.515887 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:31:15.516338 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:15.515978 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:31:15.516338 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:15.516050 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:31:16.132203 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:16.132121 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" event={"ID":"79d7a2c8-0585-48a3-88d4-d49b23672216","Type":"ContainerStarted","Data":"63338caca66d5d40f5435f9815a1df944f421544469b85e1be47e43bd2b3733e"} Oct 30 21:31:16.158323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:16.158269 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" podStartSLOduration=1.735388087 podStartE2EDuration="10.158249955s" totalImagesPullingTime="8.422861868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:16.157602148 +0000 UTC m=+144.753418829" watchObservedRunningTime="2026-10-30 21:31:16.158249955 +0000 UTC m=+144.754066617" Oct 30 21:31:16.764543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:16.763174 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:16.770192 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:16.770166 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:17.142282 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:17.142242 5801 generic.go:356] "Generic (PLEG): container finished" podID="1b9a5b39-38d0-4ef1-aa12-4d0453fdb039" containerID="a0c4b8cc6db1fb0494e4688d4c5c13db33274677cf3514345938e30ed90fb97b" exitCode=0 Oct 30 21:31:17.143879 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:17.143840 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml" event={"ID":"1b9a5b39-38d0-4ef1-aa12-4d0453fdb039","Type":"ContainerDied","Data":"a0c4b8cc6db1fb0494e4688d4c5c13db33274677cf3514345938e30ed90fb97b"} Oct 30 21:31:17.144034 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:17.143907 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:17.145991 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:17.145968 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:31:19.152942 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:19.152902 5801 generic.go:356] "Generic (PLEG): container finished" podID="67973dda-956d-4c9e-895f-02862eb2dc08" containerID="e83ba4a35f7951045a3daa2d19c8df6d8efee9f0bd09cbc3472baed38e189cda" exitCode=0 Oct 30 21:31:19.153308 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:19.153009 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw" event={"ID":"67973dda-956d-4c9e-895f-02862eb2dc08","Type":"ContainerDied","Data":"e83ba4a35f7951045a3daa2d19c8df6d8efee9f0bd09cbc3472baed38e189cda"} Oct 30 21:31:19.156313 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:19.156284 5801 generic.go:356] "Generic (PLEG): container finished" podID="20c970ef-b259-4d62-9f7c-3d80a2ef7405" containerID="67bfd6f4edcc8e57b80706fa7ff13b03f331f736a2171199a31f934b232f49ef" exitCode=0 Oct 30 21:31:19.156441 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:19.156394 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5" event={"ID":"20c970ef-b259-4d62-9f7c-3d80a2ef7405","Type":"ContainerDied","Data":"67bfd6f4edcc8e57b80706fa7ff13b03f331f736a2171199a31f934b232f49ef"} Oct 30 21:31:19.159360 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:19.159339 5801 generic.go:356] "Generic (PLEG): container finished" podID="1b9a5b39-38d0-4ef1-aa12-4d0453fdb039" containerID="3e0e7aebe7664a7dedf4b4c5131a1512355bd94596216f91042ef67c5906ebfa" exitCode=0 Oct 30 21:31:19.159527 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:19.159447 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml" event={"ID":"1b9a5b39-38d0-4ef1-aa12-4d0453fdb039","Type":"ContainerDied","Data":"3e0e7aebe7664a7dedf4b4c5131a1512355bd94596216f91042ef67c5906ebfa"} Oct 30 21:31:19.162043 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:19.162017 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn" event={"ID":"685bdc58-e232-4570-89ca-388bed110bc1","Type":"ContainerStarted","Data":"c3b9f2156800b92ccec05d8271ac4795cf69d0f2ee7060d87a7cc0f9a709d902"} Oct 30 21:31:20.167192 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:20.167155 5801 generic.go:356] "Generic (PLEG): container finished" podID="685bdc58-e232-4570-89ca-388bed110bc1" containerID="c3b9f2156800b92ccec05d8271ac4795cf69d0f2ee7060d87a7cc0f9a709d902" exitCode=0 Oct 30 21:31:20.167638 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:20.167250 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn" event={"ID":"685bdc58-e232-4570-89ca-388bed110bc1","Type":"ContainerDied","Data":"c3b9f2156800b92ccec05d8271ac4795cf69d0f2ee7060d87a7cc0f9a709d902"} Oct 30 21:31:20.169781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:20.169241 5801 generic.go:356] "Generic (PLEG): container finished" podID="b1fc8b20-9a67-4298-aa95-f07368cafb1f" containerID="02c5174d3d6bf06dcea18df1beaca6dd701793f1152fd514b38e674d1eae4522" exitCode=0 Oct 30 21:31:20.169781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:20.169336 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x" event={"ID":"b1fc8b20-9a67-4298-aa95-f07368cafb1f","Type":"ContainerDied","Data":"02c5174d3d6bf06dcea18df1beaca6dd701793f1152fd514b38e674d1eae4522"} Oct 30 21:31:21.174832 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:21.174744 5801 generic.go:356] "Generic (PLEG): container finished" podID="39b20d2f-7f91-4a12-b2a6-8045b2879a0a" containerID="58d9a160157e6b3fe24bd3c4ec0512ac47349de6be6beaba6b0211151e41d982" exitCode=0 Oct 30 21:31:21.175247 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:21.174886 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg" event={"ID":"39b20d2f-7f91-4a12-b2a6-8045b2879a0a","Type":"ContainerDied","Data":"58d9a160157e6b3fe24bd3c4ec0512ac47349de6be6beaba6b0211151e41d982"} Oct 30 21:31:21.177801 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:21.177747 5801 generic.go:356] "Generic (PLEG): container finished" podID="0781ba05-2ba2-4717-ba74-ef1244d51051" containerID="f391a39305919516257ba690e15f6dc73232e217d4895529d1893aa8d119f6ee" exitCode=0 Oct 30 21:31:21.177983 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:21.177908 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd" event={"ID":"0781ba05-2ba2-4717-ba74-ef1244d51051","Type":"ContainerDied","Data":"f391a39305919516257ba690e15f6dc73232e217d4895529d1893aa8d119f6ee"} Oct 30 21:31:21.181051 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:21.180963 5801 generic.go:356] "Generic (PLEG): container finished" podID="67973dda-956d-4c9e-895f-02862eb2dc08" containerID="a1cd9d82fbfa2155087a590d236a2d4bad5700109b6d667cd89bcd93ef18588a" exitCode=0 Oct 30 21:31:21.181119 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:21.181078 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw" event={"ID":"67973dda-956d-4c9e-895f-02862eb2dc08","Type":"ContainerDied","Data":"a1cd9d82fbfa2155087a590d236a2d4bad5700109b6d667cd89bcd93ef18588a"} Oct 30 21:31:21.184621 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:21.184591 5801 generic.go:356] "Generic (PLEG): container finished" podID="20c970ef-b259-4d62-9f7c-3d80a2ef7405" containerID="3282c2cfbf0bec8c22a29a372fc93df6cf0a01924b7113cd4551ece3fcef3b2e" exitCode=0 Oct 30 21:31:21.184621 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:21.184620 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5" event={"ID":"20c970ef-b259-4d62-9f7c-3d80a2ef7405","Type":"ContainerDied","Data":"3282c2cfbf0bec8c22a29a372fc93df6cf0a01924b7113cd4551ece3fcef3b2e"} Oct 30 21:31:22.194537 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:22.194496 5801 generic.go:356] "Generic (PLEG): container finished" podID="b1fc8b20-9a67-4298-aa95-f07368cafb1f" containerID="c7dee6393920a8384524d8a7ab3aefa3f861b0da1ca61acca2b413d54031cdb3" exitCode=0 Oct 30 21:31:22.195033 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:22.194655 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x" event={"ID":"b1fc8b20-9a67-4298-aa95-f07368cafb1f","Type":"ContainerDied","Data":"c7dee6393920a8384524d8a7ab3aefa3f861b0da1ca61acca2b413d54031cdb3"} Oct 30 21:31:22.200771 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:22.200729 5801 generic.go:356] "Generic (PLEG): container finished" podID="685bdc58-e232-4570-89ca-388bed110bc1" containerID="f3a12df80053e0f0be88daaa84e68127d725a2277befdcf4e9fc4ed09bdd7c73" exitCode=0 Oct 30 21:31:22.200877 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:22.200779 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn" event={"ID":"685bdc58-e232-4570-89ca-388bed110bc1","Type":"ContainerDied","Data":"f3a12df80053e0f0be88daaa84e68127d725a2277befdcf4e9fc4ed09bdd7c73"} Oct 30 21:31:22.702816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:22.702778 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:23.689769 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:23.689667 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:26.227266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:26.227227 5801 generic.go:356] "Generic (PLEG): container finished" podID="39b20d2f-7f91-4a12-b2a6-8045b2879a0a" containerID="32c17a6de8ca7d320c9784017ea4b8a27919b5ccbb74c0d1e2b34e68e0756904" exitCode=0 Oct 30 21:31:26.227808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:26.227316 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg" event={"ID":"39b20d2f-7f91-4a12-b2a6-8045b2879a0a","Type":"ContainerDied","Data":"32c17a6de8ca7d320c9784017ea4b8a27919b5ccbb74c0d1e2b34e68e0756904"} Oct 30 21:31:26.233019 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:26.232802 5801 generic.go:356] "Generic (PLEG): container finished" podID="0781ba05-2ba2-4717-ba74-ef1244d51051" containerID="1170119d4b1052a6224fe0fe6a97f7c2ac9fb90361d8a869f3396a1cafbb3dbd" exitCode=0 Oct 30 21:31:26.233019 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:26.232929 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd" event={"ID":"0781ba05-2ba2-4717-ba74-ef1244d51051","Type":"ContainerDied","Data":"1170119d4b1052a6224fe0fe6a97f7c2ac9fb90361d8a869f3396a1cafbb3dbd"} Oct 30 21:31:30.292306 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:31:30.292216 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:31:32.275307 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:32.274687 5801 generic.go:356] "Generic (PLEG): container finished" podID="1b9a5b39-38d0-4ef1-aa12-4d0453fdb039" containerID="9ee8e2b783d6350243166eb485f8362c0b33929b0fedd8583a18dc775bed52f0" exitCode=0 Oct 30 21:31:32.275307 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:32.274834 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml" event={"ID":"1b9a5b39-38d0-4ef1-aa12-4d0453fdb039","Type":"ContainerDied","Data":"9ee8e2b783d6350243166eb485f8362c0b33929b0fedd8583a18dc775bed52f0"} Oct 30 21:31:33.469324 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:33.469173 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml" Oct 30 21:31:33.579320 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:33.579284 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b9a5b39-38d0-4ef1-aa12-4d0453fdb039-kube-api-access-4krsr\" (UniqueName: \"kubernetes.io/projected/1b9a5b39-38d0-4ef1-aa12-4d0453fdb039-kube-api-access-4krsr\") pod \"1b9a5b39-38d0-4ef1-aa12-4d0453fdb039\" (UID: \"1b9a5b39-38d0-4ef1-aa12-4d0453fdb039\") " Oct 30 21:31:33.579524 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:33.579414 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9a5b39-38d0-4ef1-aa12-4d0453fdb039-util\" (UniqueName: \"kubernetes.io/empty-dir/1b9a5b39-38d0-4ef1-aa12-4d0453fdb039-util\") pod \"1b9a5b39-38d0-4ef1-aa12-4d0453fdb039\" (UID: \"1b9a5b39-38d0-4ef1-aa12-4d0453fdb039\") " Oct 30 21:31:33.579524 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:33.579457 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9a5b39-38d0-4ef1-aa12-4d0453fdb039-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b9a5b39-38d0-4ef1-aa12-4d0453fdb039-bundle\") pod \"1b9a5b39-38d0-4ef1-aa12-4d0453fdb039\" (UID: \"1b9a5b39-38d0-4ef1-aa12-4d0453fdb039\") " Oct 30 21:31:33.580192 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:33.580164 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9a5b39-38d0-4ef1-aa12-4d0453fdb039-bundle" pod "1b9a5b39-38d0-4ef1-aa12-4d0453fdb039" (UID: "1b9a5b39-38d0-4ef1-aa12-4d0453fdb039"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:33.585855 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:33.585824 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9a5b39-38d0-4ef1-aa12-4d0453fdb039-util" pod "1b9a5b39-38d0-4ef1-aa12-4d0453fdb039" (UID: "1b9a5b39-38d0-4ef1-aa12-4d0453fdb039"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:33.591020 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:33.590974 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9a5b39-38d0-4ef1-aa12-4d0453fdb039-kube-api-access-4krsr" pod "1b9a5b39-38d0-4ef1-aa12-4d0453fdb039" (UID: "1b9a5b39-38d0-4ef1-aa12-4d0453fdb039"). InnerVolumeSpecName "kube-api-access-4krsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:33.681035 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:33.680961 5801 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b9a5b39-38d0-4ef1-aa12-4d0453fdb039-util\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:33.681035 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:33.680998 5801 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b9a5b39-38d0-4ef1-aa12-4d0453fdb039-bundle\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:33.681035 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:33.681013 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4krsr\" (UniqueName: \"kubernetes.io/projected/1b9a5b39-38d0-4ef1-aa12-4d0453fdb039-kube-api-access-4krsr\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:34.297919 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:34.294441 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml" event={"ID":"1b9a5b39-38d0-4ef1-aa12-4d0453fdb039","Type":"ContainerDied","Data":"d151f1d56874b5f79d3bdd3eed73d0fca080e6ef96f6d3f3e3307ca80f9f2b44"} Oct 30 21:31:34.297919 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:34.294505 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d151f1d56874b5f79d3bdd3eed73d0fca080e6ef96f6d3f3e3307ca80f9f2b44" Oct 30 21:31:34.297919 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:34.294603 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vkpml" Oct 30 21:31:35.302663 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:35.302629 5801 generic.go:356] "Generic (PLEG): container finished" podID="67973dda-956d-4c9e-895f-02862eb2dc08" containerID="3ad0ebf82bcfa32c1e9ae18c1aff06fbfabaa1a93000b9e19d081aba08c076a4" exitCode=0 Oct 30 21:31:35.303068 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:35.302696 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw" event={"ID":"67973dda-956d-4c9e-895f-02862eb2dc08","Type":"ContainerDied","Data":"3ad0ebf82bcfa32c1e9ae18c1aff06fbfabaa1a93000b9e19d081aba08c076a4"} Oct 30 21:31:36.308261 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:36.308230 5801 generic.go:356] "Generic (PLEG): container finished" podID="20c970ef-b259-4d62-9f7c-3d80a2ef7405" containerID="79d97a5c09f20cafcd0fe7f96d3b3dac57c78f9df018d88a51a4ff724c613eae" exitCode=0 Oct 30 21:31:36.308665 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:36.308317 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5" event={"ID":"20c970ef-b259-4d62-9f7c-3d80a2ef7405","Type":"ContainerDied","Data":"79d97a5c09f20cafcd0fe7f96d3b3dac57c78f9df018d88a51a4ff724c613eae"} Oct 30 21:31:36.310287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:36.310255 5801 generic.go:356] "Generic (PLEG): container finished" podID="39b20d2f-7f91-4a12-b2a6-8045b2879a0a" containerID="3672e8348547d7095a4548277557ce5cf7168d291c8bf4140ca5d303cc4b8db7" exitCode=0 Oct 30 21:31:36.310399 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:36.310312 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg" event={"ID":"39b20d2f-7f91-4a12-b2a6-8045b2879a0a","Type":"ContainerDied","Data":"3672e8348547d7095a4548277557ce5cf7168d291c8bf4140ca5d303cc4b8db7"} Oct 30 21:31:36.312259 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:36.312241 5801 generic.go:356] "Generic (PLEG): container finished" podID="0781ba05-2ba2-4717-ba74-ef1244d51051" containerID="2ed5bf6b74d496be069551a5b1cd5d8f720eb31bf59183f6f104339d4a5378b1" exitCode=0 Oct 30 21:31:36.312384 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:36.312335 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd" event={"ID":"0781ba05-2ba2-4717-ba74-ef1244d51051","Type":"ContainerDied","Data":"2ed5bf6b74d496be069551a5b1cd5d8f720eb31bf59183f6f104339d4a5378b1"} Oct 30 21:31:36.314359 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:36.314337 5801 generic.go:356] "Generic (PLEG): container finished" podID="685bdc58-e232-4570-89ca-388bed110bc1" containerID="911001c4d251473ac454b3dd8de3ea3c18a23fc74a4a3f7e934178221ab1242e" exitCode=0 Oct 30 21:31:36.314440 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:36.314415 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn" event={"ID":"685bdc58-e232-4570-89ca-388bed110bc1","Type":"ContainerDied","Data":"911001c4d251473ac454b3dd8de3ea3c18a23fc74a4a3f7e934178221ab1242e"} Oct 30 21:31:36.316268 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:36.316249 5801 generic.go:356] "Generic (PLEG): container finished" podID="b1fc8b20-9a67-4298-aa95-f07368cafb1f" containerID="545f0436f625c76e86fe9157056beaaa532426ab87fffd0a01f9dd1e13affe98" exitCode=0 Oct 30 21:31:36.316363 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:36.316311 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x" event={"ID":"b1fc8b20-9a67-4298-aa95-f07368cafb1f","Type":"ContainerDied","Data":"545f0436f625c76e86fe9157056beaaa532426ab87fffd0a01f9dd1e13affe98"} Oct 30 21:31:36.441941 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:36.441915 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw" Oct 30 21:31:36.505146 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:36.505110 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67973dda-956d-4c9e-895f-02862eb2dc08-kube-api-access-cmlc2\" (UniqueName: \"kubernetes.io/projected/67973dda-956d-4c9e-895f-02862eb2dc08-kube-api-access-cmlc2\") pod \"67973dda-956d-4c9e-895f-02862eb2dc08\" (UID: \"67973dda-956d-4c9e-895f-02862eb2dc08\") " Oct 30 21:31:36.505305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:36.505154 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67973dda-956d-4c9e-895f-02862eb2dc08-util\" (UniqueName: \"kubernetes.io/empty-dir/67973dda-956d-4c9e-895f-02862eb2dc08-util\") pod \"67973dda-956d-4c9e-895f-02862eb2dc08\" (UID: \"67973dda-956d-4c9e-895f-02862eb2dc08\") " Oct 30 21:31:36.505305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:36.505203 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67973dda-956d-4c9e-895f-02862eb2dc08-bundle\" (UniqueName: \"kubernetes.io/empty-dir/67973dda-956d-4c9e-895f-02862eb2dc08-bundle\") pod \"67973dda-956d-4c9e-895f-02862eb2dc08\" (UID: \"67973dda-956d-4c9e-895f-02862eb2dc08\") " Oct 30 21:31:36.505896 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:36.505863 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67973dda-956d-4c9e-895f-02862eb2dc08-bundle" pod "67973dda-956d-4c9e-895f-02862eb2dc08" (UID: "67973dda-956d-4c9e-895f-02862eb2dc08"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:36.507173 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:36.507149 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67973dda-956d-4c9e-895f-02862eb2dc08-kube-api-access-cmlc2" pod "67973dda-956d-4c9e-895f-02862eb2dc08" (UID: "67973dda-956d-4c9e-895f-02862eb2dc08"). InnerVolumeSpecName "kube-api-access-cmlc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:36.510556 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:36.510531 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67973dda-956d-4c9e-895f-02862eb2dc08-util" pod "67973dda-956d-4c9e-895f-02862eb2dc08" (UID: "67973dda-956d-4c9e-895f-02862eb2dc08"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:36.606046 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:36.605955 5801 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67973dda-956d-4c9e-895f-02862eb2dc08-bundle\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:36.606046 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:36.605989 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmlc2\" (UniqueName: \"kubernetes.io/projected/67973dda-956d-4c9e-895f-02862eb2dc08-kube-api-access-cmlc2\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:36.606046 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:36.606003 5801 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67973dda-956d-4c9e-895f-02862eb2dc08-util\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:37.322265 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.322221 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw" event={"ID":"67973dda-956d-4c9e-895f-02862eb2dc08","Type":"ContainerDied","Data":"fdbc47d7c67a500ef1dabdcc7aa26bb0e9533cd23ee15a50516586fd48e625d9"} Oct 30 21:31:37.322714 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.322298 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdbc47d7c67a500ef1dabdcc7aa26bb0e9533cd23ee15a50516586fd48e625d9" Oct 30 21:31:37.322714 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.322240 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847asx9nw" Oct 30 21:31:37.460280 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.460253 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd" Oct 30 21:31:37.488505 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.488481 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5" Oct 30 21:31:37.500994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.500972 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg" Oct 30 21:31:37.513738 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.513712 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0781ba05-2ba2-4717-ba74-ef1244d51051-kube-api-access-c8mdr\" (UniqueName: \"kubernetes.io/projected/0781ba05-2ba2-4717-ba74-ef1244d51051-kube-api-access-c8mdr\") pod \"0781ba05-2ba2-4717-ba74-ef1244d51051\" (UID: \"0781ba05-2ba2-4717-ba74-ef1244d51051\") " Oct 30 21:31:37.513885 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.513821 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0781ba05-2ba2-4717-ba74-ef1244d51051-util\" (UniqueName: \"kubernetes.io/empty-dir/0781ba05-2ba2-4717-ba74-ef1244d51051-util\") pod \"0781ba05-2ba2-4717-ba74-ef1244d51051\" (UID: \"0781ba05-2ba2-4717-ba74-ef1244d51051\") " Oct 30 21:31:37.513885 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.513864 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0781ba05-2ba2-4717-ba74-ef1244d51051-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0781ba05-2ba2-4717-ba74-ef1244d51051-bundle\") pod \"0781ba05-2ba2-4717-ba74-ef1244d51051\" (UID: \"0781ba05-2ba2-4717-ba74-ef1244d51051\") " Oct 30 21:31:37.514399 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.514370 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0781ba05-2ba2-4717-ba74-ef1244d51051-bundle" pod "0781ba05-2ba2-4717-ba74-ef1244d51051" (UID: "0781ba05-2ba2-4717-ba74-ef1244d51051"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:37.519303 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.519279 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0781ba05-2ba2-4717-ba74-ef1244d51051-kube-api-access-c8mdr" pod "0781ba05-2ba2-4717-ba74-ef1244d51051" (UID: "0781ba05-2ba2-4717-ba74-ef1244d51051"). InnerVolumeSpecName "kube-api-access-c8mdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:37.520663 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.520636 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0781ba05-2ba2-4717-ba74-ef1244d51051-util" pod "0781ba05-2ba2-4717-ba74-ef1244d51051" (UID: "0781ba05-2ba2-4717-ba74-ef1244d51051"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:37.535545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.535432 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x" Oct 30 21:31:37.586727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.586705 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn" Oct 30 21:31:37.615153 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.615124 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1fc8b20-9a67-4298-aa95-f07368cafb1f-util\" (UniqueName: \"kubernetes.io/empty-dir/b1fc8b20-9a67-4298-aa95-f07368cafb1f-util\") pod \"b1fc8b20-9a67-4298-aa95-f07368cafb1f\" (UID: \"b1fc8b20-9a67-4298-aa95-f07368cafb1f\") " Oct 30 21:31:37.615301 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.615173 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1fc8b20-9a67-4298-aa95-f07368cafb1f-kube-api-access-l2xf5\" (UniqueName: \"kubernetes.io/projected/b1fc8b20-9a67-4298-aa95-f07368cafb1f-kube-api-access-l2xf5\") pod \"b1fc8b20-9a67-4298-aa95-f07368cafb1f\" (UID: \"b1fc8b20-9a67-4298-aa95-f07368cafb1f\") " Oct 30 21:31:37.615301 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.615208 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b20d2f-7f91-4a12-b2a6-8045b2879a0a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/39b20d2f-7f91-4a12-b2a6-8045b2879a0a-bundle\") pod \"39b20d2f-7f91-4a12-b2a6-8045b2879a0a\" (UID: \"39b20d2f-7f91-4a12-b2a6-8045b2879a0a\") " Oct 30 21:31:37.615301 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.615229 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20c970ef-b259-4d62-9f7c-3d80a2ef7405-bundle\" (UniqueName: \"kubernetes.io/empty-dir/20c970ef-b259-4d62-9f7c-3d80a2ef7405-bundle\") pod \"20c970ef-b259-4d62-9f7c-3d80a2ef7405\" (UID: \"20c970ef-b259-4d62-9f7c-3d80a2ef7405\") " Oct 30 21:31:37.615301 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.615248 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39b20d2f-7f91-4a12-b2a6-8045b2879a0a-kube-api-access-r2zqc\" (UniqueName: \"kubernetes.io/projected/39b20d2f-7f91-4a12-b2a6-8045b2879a0a-kube-api-access-r2zqc\") pod \"39b20d2f-7f91-4a12-b2a6-8045b2879a0a\" (UID: \"39b20d2f-7f91-4a12-b2a6-8045b2879a0a\") " Oct 30 21:31:37.615301 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.615290 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c970ef-b259-4d62-9f7c-3d80a2ef7405-kube-api-access-6gk5r\" (UniqueName: \"kubernetes.io/projected/20c970ef-b259-4d62-9f7c-3d80a2ef7405-kube-api-access-6gk5r\") pod \"20c970ef-b259-4d62-9f7c-3d80a2ef7405\" (UID: \"20c970ef-b259-4d62-9f7c-3d80a2ef7405\") " Oct 30 21:31:37.615581 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.615348 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1fc8b20-9a67-4298-aa95-f07368cafb1f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1fc8b20-9a67-4298-aa95-f07368cafb1f-bundle\") pod \"b1fc8b20-9a67-4298-aa95-f07368cafb1f\" (UID: \"b1fc8b20-9a67-4298-aa95-f07368cafb1f\") " Oct 30 21:31:37.615581 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.615388 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20c970ef-b259-4d62-9f7c-3d80a2ef7405-util\" (UniqueName: \"kubernetes.io/empty-dir/20c970ef-b259-4d62-9f7c-3d80a2ef7405-util\") pod \"20c970ef-b259-4d62-9f7c-3d80a2ef7405\" (UID: \"20c970ef-b259-4d62-9f7c-3d80a2ef7405\") " Oct 30 21:31:37.615581 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.615417 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b20d2f-7f91-4a12-b2a6-8045b2879a0a-util\" (UniqueName: \"kubernetes.io/empty-dir/39b20d2f-7f91-4a12-b2a6-8045b2879a0a-util\") pod \"39b20d2f-7f91-4a12-b2a6-8045b2879a0a\" (UID: \"39b20d2f-7f91-4a12-b2a6-8045b2879a0a\") " Oct 30 21:31:37.615734 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.615681 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8mdr\" (UniqueName: \"kubernetes.io/projected/0781ba05-2ba2-4717-ba74-ef1244d51051-kube-api-access-c8mdr\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:37.615734 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.615703 5801 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0781ba05-2ba2-4717-ba74-ef1244d51051-util\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:37.615734 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.615718 5801 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0781ba05-2ba2-4717-ba74-ef1244d51051-bundle\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:37.621706 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.615938 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c970ef-b259-4d62-9f7c-3d80a2ef7405-bundle" pod "20c970ef-b259-4d62-9f7c-3d80a2ef7405" (UID: "20c970ef-b259-4d62-9f7c-3d80a2ef7405"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:37.621706 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.615946 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1fc8b20-9a67-4298-aa95-f07368cafb1f-bundle" pod "b1fc8b20-9a67-4298-aa95-f07368cafb1f" (UID: "b1fc8b20-9a67-4298-aa95-f07368cafb1f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:37.621706 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.616466 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b20d2f-7f91-4a12-b2a6-8045b2879a0a-bundle" pod "39b20d2f-7f91-4a12-b2a6-8045b2879a0a" (UID: "39b20d2f-7f91-4a12-b2a6-8045b2879a0a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:37.621706 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.618348 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1fc8b20-9a67-4298-aa95-f07368cafb1f-kube-api-access-l2xf5" pod "b1fc8b20-9a67-4298-aa95-f07368cafb1f" (UID: "b1fc8b20-9a67-4298-aa95-f07368cafb1f"). InnerVolumeSpecName "kube-api-access-l2xf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:37.621706 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.618606 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b20d2f-7f91-4a12-b2a6-8045b2879a0a-kube-api-access-r2zqc" pod "39b20d2f-7f91-4a12-b2a6-8045b2879a0a" (UID: "39b20d2f-7f91-4a12-b2a6-8045b2879a0a"). InnerVolumeSpecName "kube-api-access-r2zqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:37.621706 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.619516 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c970ef-b259-4d62-9f7c-3d80a2ef7405-kube-api-access-6gk5r" pod "20c970ef-b259-4d62-9f7c-3d80a2ef7405" (UID: "20c970ef-b259-4d62-9f7c-3d80a2ef7405"). InnerVolumeSpecName "kube-api-access-6gk5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:37.621706 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.621668 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1fc8b20-9a67-4298-aa95-f07368cafb1f-util" pod "b1fc8b20-9a67-4298-aa95-f07368cafb1f" (UID: "b1fc8b20-9a67-4298-aa95-f07368cafb1f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:37.622902 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.622869 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b20d2f-7f91-4a12-b2a6-8045b2879a0a-util" pod "39b20d2f-7f91-4a12-b2a6-8045b2879a0a" (UID: "39b20d2f-7f91-4a12-b2a6-8045b2879a0a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:37.623297 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.623275 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c970ef-b259-4d62-9f7c-3d80a2ef7405-util" pod "20c970ef-b259-4d62-9f7c-3d80a2ef7405" (UID: "20c970ef-b259-4d62-9f7c-3d80a2ef7405"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:37.716800 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.716739 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/685bdc58-e232-4570-89ca-388bed110bc1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/685bdc58-e232-4570-89ca-388bed110bc1-bundle\") pod \"685bdc58-e232-4570-89ca-388bed110bc1\" (UID: \"685bdc58-e232-4570-89ca-388bed110bc1\") " Oct 30 21:31:37.716800 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.716804 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/685bdc58-e232-4570-89ca-388bed110bc1-util\" (UniqueName: \"kubernetes.io/empty-dir/685bdc58-e232-4570-89ca-388bed110bc1-util\") pod \"685bdc58-e232-4570-89ca-388bed110bc1\" (UID: \"685bdc58-e232-4570-89ca-388bed110bc1\") " Oct 30 21:31:37.716984 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.716833 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/685bdc58-e232-4570-89ca-388bed110bc1-kube-api-access-8zjq2\" (UniqueName: \"kubernetes.io/projected/685bdc58-e232-4570-89ca-388bed110bc1-kube-api-access-8zjq2\") pod \"685bdc58-e232-4570-89ca-388bed110bc1\" (UID: \"685bdc58-e232-4570-89ca-388bed110bc1\") " Oct 30 21:31:37.717045 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.717028 5801 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1fc8b20-9a67-4298-aa95-f07368cafb1f-util\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:37.717079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.717054 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2xf5\" (UniqueName: \"kubernetes.io/projected/b1fc8b20-9a67-4298-aa95-f07368cafb1f-kube-api-access-l2xf5\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:37.717079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.717068 5801 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39b20d2f-7f91-4a12-b2a6-8045b2879a0a-bundle\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:37.717160 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.717082 5801 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20c970ef-b259-4d62-9f7c-3d80a2ef7405-bundle\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:37.717160 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.717096 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2zqc\" (UniqueName: \"kubernetes.io/projected/39b20d2f-7f91-4a12-b2a6-8045b2879a0a-kube-api-access-r2zqc\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:37.717160 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.717113 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gk5r\" (UniqueName: \"kubernetes.io/projected/20c970ef-b259-4d62-9f7c-3d80a2ef7405-kube-api-access-6gk5r\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:37.717160 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.717127 5801 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1fc8b20-9a67-4298-aa95-f07368cafb1f-bundle\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:37.717160 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.717141 5801 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20c970ef-b259-4d62-9f7c-3d80a2ef7405-util\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:37.717160 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.717152 5801 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39b20d2f-7f91-4a12-b2a6-8045b2879a0a-util\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:37.717323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.717190 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/685bdc58-e232-4570-89ca-388bed110bc1-bundle" pod "685bdc58-e232-4570-89ca-388bed110bc1" (UID: "685bdc58-e232-4570-89ca-388bed110bc1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:37.718809 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.718784 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/685bdc58-e232-4570-89ca-388bed110bc1-kube-api-access-8zjq2" pod "685bdc58-e232-4570-89ca-388bed110bc1" (UID: "685bdc58-e232-4570-89ca-388bed110bc1"). InnerVolumeSpecName "kube-api-access-8zjq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:37.720689 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.720670 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/685bdc58-e232-4570-89ca-388bed110bc1-util" pod "685bdc58-e232-4570-89ca-388bed110bc1" (UID: "685bdc58-e232-4570-89ca-388bed110bc1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:37.818287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.818245 5801 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/685bdc58-e232-4570-89ca-388bed110bc1-bundle\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:37.818287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.818283 5801 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/685bdc58-e232-4570-89ca-388bed110bc1-util\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:37.818287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:37.818293 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zjq2\" (UniqueName: \"kubernetes.io/projected/685bdc58-e232-4570-89ca-388bed110bc1-kube-api-access-8zjq2\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:38.328093 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:38.328054 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5" event={"ID":"20c970ef-b259-4d62-9f7c-3d80a2ef7405","Type":"ContainerDied","Data":"33b7870c91f30c877a3fb74c06b7a305cfc8955ba7cae21cfbd30c0b5c557756"} Oct 30 21:31:38.328548 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:38.328109 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33b7870c91f30c877a3fb74c06b7a305cfc8955ba7cae21cfbd30c0b5c557756" Oct 30 21:31:38.328548 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:38.328068 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqf2h5" Oct 30 21:31:38.330000 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:38.329978 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg" Oct 30 21:31:38.330130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:38.329977 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7fqg" event={"ID":"39b20d2f-7f91-4a12-b2a6-8045b2879a0a","Type":"ContainerDied","Data":"bd9eddf466486bf5cfaf34061a6e97488a6a555e417beb1526163b0d1fb6ab4f"} Oct 30 21:31:38.330130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:38.330085 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd9eddf466486bf5cfaf34061a6e97488a6a555e417beb1526163b0d1fb6ab4f" Oct 30 21:31:38.331911 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:38.331890 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd" Oct 30 21:31:38.332027 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:38.331911 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442zrxd" event={"ID":"0781ba05-2ba2-4717-ba74-ef1244d51051","Type":"ContainerDied","Data":"d87f292196aae977f07b676bc4f39c1051702578685b9c7c9e4dd155ac7f370d"} Oct 30 21:31:38.332027 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:38.331946 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d87f292196aae977f07b676bc4f39c1051702578685b9c7c9e4dd155ac7f370d" Oct 30 21:31:38.334063 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:38.334042 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn" Oct 30 21:31:38.334344 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:38.334090 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tppqn" event={"ID":"685bdc58-e232-4570-89ca-388bed110bc1","Type":"ContainerDied","Data":"e2bdfceedbc58d1ac4852c13f1ea45cef8ee5bb428a9eafabb14ebabbe67269d"} Oct 30 21:31:38.334344 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:38.334286 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2bdfceedbc58d1ac4852c13f1ea45cef8ee5bb428a9eafabb14ebabbe67269d" Oct 30 21:31:38.336157 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:38.336133 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x" event={"ID":"b1fc8b20-9a67-4298-aa95-f07368cafb1f","Type":"ContainerDied","Data":"c914a0c9392f537ec9aab97814fedb3aa3e545789abdbd2d17a9d5946e71d207"} Oct 30 21:31:38.336274 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:38.336171 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s7z5x" Oct 30 21:31:38.336274 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:38.336172 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c914a0c9392f537ec9aab97814fedb3aa3e545789abdbd2d17a9d5946e71d207" Oct 30 21:31:40.380434 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:40.380400 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:40.480281 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:40.480242 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:41.139928 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.139885 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:41.280809 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.280778 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb"] Oct 30 21:31:41.281166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281149 5801 cpu_manager.go:414] "containerMap: removing container" podUID="20c970ef-b259-4d62-9f7c-3d80a2ef7405" containerName="pull" Oct 30 21:31:41.281253 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281168 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c970ef-b259-4d62-9f7c-3d80a2ef7405" containerName="pull" Oct 30 21:31:41.281253 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281191 5801 cpu_manager.go:414] "containerMap: removing container" podUID="685bdc58-e232-4570-89ca-388bed110bc1" containerName="pull" Oct 30 21:31:41.281253 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281199 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="685bdc58-e232-4570-89ca-388bed110bc1" containerName="pull" Oct 30 21:31:41.281253 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281209 5801 cpu_manager.go:414] "containerMap: removing container" podUID="39b20d2f-7f91-4a12-b2a6-8045b2879a0a" containerName="pull" Oct 30 21:31:41.281253 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281217 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39b20d2f-7f91-4a12-b2a6-8045b2879a0a" containerName="pull" Oct 30 21:31:41.281253 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281225 5801 cpu_manager.go:414] "containerMap: removing container" podUID="39b20d2f-7f91-4a12-b2a6-8045b2879a0a" containerName="extract" Oct 30 21:31:41.281253 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281232 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39b20d2f-7f91-4a12-b2a6-8045b2879a0a" containerName="extract" Oct 30 21:31:41.281253 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281252 5801 cpu_manager.go:414] "containerMap: removing container" podUID="1b9a5b39-38d0-4ef1-aa12-4d0453fdb039" containerName="pull" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281259 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b9a5b39-38d0-4ef1-aa12-4d0453fdb039" containerName="pull" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281271 5801 cpu_manager.go:414] "containerMap: removing container" podUID="b1fc8b20-9a67-4298-aa95-f07368cafb1f" containerName="pull" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281278 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1fc8b20-9a67-4298-aa95-f07368cafb1f" containerName="pull" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281289 5801 cpu_manager.go:414] "containerMap: removing container" podUID="1b9a5b39-38d0-4ef1-aa12-4d0453fdb039" containerName="extract" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281296 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b9a5b39-38d0-4ef1-aa12-4d0453fdb039" containerName="extract" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281305 5801 cpu_manager.go:414] "containerMap: removing container" podUID="20c970ef-b259-4d62-9f7c-3d80a2ef7405" containerName="extract" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281312 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c970ef-b259-4d62-9f7c-3d80a2ef7405" containerName="extract" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281324 5801 cpu_manager.go:414] "containerMap: removing container" podUID="0781ba05-2ba2-4717-ba74-ef1244d51051" containerName="pull" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281330 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0781ba05-2ba2-4717-ba74-ef1244d51051" containerName="pull" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281343 5801 cpu_manager.go:414] "containerMap: removing container" podUID="67973dda-956d-4c9e-895f-02862eb2dc08" containerName="extract" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281350 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67973dda-956d-4c9e-895f-02862eb2dc08" containerName="extract" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281365 5801 cpu_manager.go:414] "containerMap: removing container" podUID="685bdc58-e232-4570-89ca-388bed110bc1" containerName="extract" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281372 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="685bdc58-e232-4570-89ca-388bed110bc1" containerName="extract" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281384 5801 cpu_manager.go:414] "containerMap: removing container" podUID="1b9a5b39-38d0-4ef1-aa12-4d0453fdb039" containerName="util" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281392 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b9a5b39-38d0-4ef1-aa12-4d0453fdb039" containerName="util" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281403 5801 cpu_manager.go:414] "containerMap: removing container" podUID="0781ba05-2ba2-4717-ba74-ef1244d51051" containerName="util" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281411 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0781ba05-2ba2-4717-ba74-ef1244d51051" containerName="util" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281420 5801 cpu_manager.go:414] "containerMap: removing container" podUID="0781ba05-2ba2-4717-ba74-ef1244d51051" containerName="extract" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281427 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0781ba05-2ba2-4717-ba74-ef1244d51051" containerName="extract" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281438 5801 cpu_manager.go:414] "containerMap: removing container" podUID="b1fc8b20-9a67-4298-aa95-f07368cafb1f" containerName="util" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281445 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1fc8b20-9a67-4298-aa95-f07368cafb1f" containerName="util" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281464 5801 cpu_manager.go:414] "containerMap: removing container" podUID="39b20d2f-7f91-4a12-b2a6-8045b2879a0a" containerName="util" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281471 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39b20d2f-7f91-4a12-b2a6-8045b2879a0a" containerName="util" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281484 5801 cpu_manager.go:414] "containerMap: removing container" podUID="20c970ef-b259-4d62-9f7c-3d80a2ef7405" containerName="util" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281491 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c970ef-b259-4d62-9f7c-3d80a2ef7405" containerName="util" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281503 5801 cpu_manager.go:414] "containerMap: removing container" podUID="67973dda-956d-4c9e-895f-02862eb2dc08" containerName="pull" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281509 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67973dda-956d-4c9e-895f-02862eb2dc08" containerName="pull" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281520 5801 cpu_manager.go:414] "containerMap: removing container" podUID="b1fc8b20-9a67-4298-aa95-f07368cafb1f" containerName="extract" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281527 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1fc8b20-9a67-4298-aa95-f07368cafb1f" containerName="extract" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281540 5801 cpu_manager.go:414] "containerMap: removing container" podUID="67973dda-956d-4c9e-895f-02862eb2dc08" containerName="util" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281546 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67973dda-956d-4c9e-895f-02862eb2dc08" containerName="util" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281555 5801 cpu_manager.go:414] "containerMap: removing container" podUID="685bdc58-e232-4570-89ca-388bed110bc1" containerName="util" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281562 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="685bdc58-e232-4570-89ca-388bed110bc1" containerName="util" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281624 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="685bdc58-e232-4570-89ca-388bed110bc1" containerName="extract" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281636 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="0781ba05-2ba2-4717-ba74-ef1244d51051" containerName="extract" Oct 30 21:31:41.281635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281646 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1fc8b20-9a67-4298-aa95-f07368cafb1f" containerName="extract" Oct 30 21:31:41.283122 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281655 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="39b20d2f-7f91-4a12-b2a6-8045b2879a0a" containerName="extract" Oct 30 21:31:41.283122 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281665 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b9a5b39-38d0-4ef1-aa12-4d0453fdb039" containerName="extract" Oct 30 21:31:41.283122 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281673 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c970ef-b259-4d62-9f7c-3d80a2ef7405" containerName="extract" Oct 30 21:31:41.283122 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.281682 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="67973dda-956d-4c9e-895f-02862eb2dc08" containerName="extract" Oct 30 21:31:41.286008 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.285988 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" Oct 30 21:31:41.292321 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.292251 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 21:31:41.292636 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.292599 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 21:31:41.293407 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.293389 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 21:31:41.293488 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.293412 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 21:31:41.293488 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.293463 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jbnqm\"" Oct 30 21:31:41.293488 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.293467 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 21:31:41.315696 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.315673 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb"] Oct 30 21:31:41.443542 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.443464 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:41.446537 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.446515 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5b8871bf-6ac4-4d11-a064-f0a762af556d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-nqfhb\" (UID: \"5b8871bf-6ac4-4d11-a064-f0a762af556d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" Oct 30 21:31:41.446666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.446546 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5b8871bf-6ac4-4d11-a064-f0a762af556d-manifests\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-nqfhb\" (UID: \"5b8871bf-6ac4-4d11-a064-f0a762af556d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" Oct 30 21:31:41.446666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.446574 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b8871bf-6ac4-4d11-a064-f0a762af556d-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-nqfhb\" (UID: \"5b8871bf-6ac4-4d11-a064-f0a762af556d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" Oct 30 21:31:41.446791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.446731 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b8871bf-6ac4-4d11-a064-f0a762af556d-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-nqfhb\" (UID: \"5b8871bf-6ac4-4d11-a064-f0a762af556d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" Oct 30 21:31:41.446855 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.446800 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnwxz\" (UniqueName: \"kubernetes.io/projected/5b8871bf-6ac4-4d11-a064-f0a762af556d-kube-api-access-pnwxz\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-nqfhb\" (UID: \"5b8871bf-6ac4-4d11-a064-f0a762af556d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" Oct 30 21:31:41.446855 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.446833 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b8871bf-6ac4-4d11-a064-f0a762af556d-tmp\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-nqfhb\" (UID: \"5b8871bf-6ac4-4d11-a064-f0a762af556d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" Oct 30 21:31:41.547442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.547415 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5b8871bf-6ac4-4d11-a064-f0a762af556d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-nqfhb\" (UID: \"5b8871bf-6ac4-4d11-a064-f0a762af556d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" Oct 30 21:31:41.547591 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.547444 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5b8871bf-6ac4-4d11-a064-f0a762af556d-manifests\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-nqfhb\" (UID: \"5b8871bf-6ac4-4d11-a064-f0a762af556d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" Oct 30 21:31:41.547591 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.547471 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b8871bf-6ac4-4d11-a064-f0a762af556d-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-nqfhb\" (UID: \"5b8871bf-6ac4-4d11-a064-f0a762af556d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" Oct 30 21:31:41.547591 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.547508 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b8871bf-6ac4-4d11-a064-f0a762af556d-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-nqfhb\" (UID: \"5b8871bf-6ac4-4d11-a064-f0a762af556d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" Oct 30 21:31:41.547591 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.547531 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnwxz\" (UniqueName: \"kubernetes.io/projected/5b8871bf-6ac4-4d11-a064-f0a762af556d-kube-api-access-pnwxz\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-nqfhb\" (UID: \"5b8871bf-6ac4-4d11-a064-f0a762af556d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" Oct 30 21:31:41.547591 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.547553 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b8871bf-6ac4-4d11-a064-f0a762af556d-tmp\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-nqfhb\" (UID: \"5b8871bf-6ac4-4d11-a064-f0a762af556d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" Oct 30 21:31:41.547923 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.547900 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5b8871bf-6ac4-4d11-a064-f0a762af556d-manifests\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-nqfhb\" (UID: \"5b8871bf-6ac4-4d11-a064-f0a762af556d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" Oct 30 21:31:41.549981 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.549945 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b8871bf-6ac4-4d11-a064-f0a762af556d-tmp\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-nqfhb\" (UID: \"5b8871bf-6ac4-4d11-a064-f0a762af556d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" Oct 30 21:31:41.550109 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.550016 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5b8871bf-6ac4-4d11-a064-f0a762af556d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-nqfhb\" (UID: \"5b8871bf-6ac4-4d11-a064-f0a762af556d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" Oct 30 21:31:41.550109 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.550029 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b8871bf-6ac4-4d11-a064-f0a762af556d-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-nqfhb\" (UID: \"5b8871bf-6ac4-4d11-a064-f0a762af556d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" Oct 30 21:31:41.550208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.550121 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b8871bf-6ac4-4d11-a064-f0a762af556d-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-nqfhb\" (UID: \"5b8871bf-6ac4-4d11-a064-f0a762af556d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" Oct 30 21:31:41.563875 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.563851 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnwxz\" (UniqueName: \"kubernetes.io/projected/5b8871bf-6ac4-4d11-a064-f0a762af556d-kube-api-access-pnwxz\") pod \"opendatahub-operator-controller-manager-6bbc596d7f-nqfhb\" (UID: \"5b8871bf-6ac4-4d11-a064-f0a762af556d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" Oct 30 21:31:41.582854 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.582832 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:41.600278 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.600254 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" Oct 30 21:31:41.689360 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.689329 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:41.896259 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:41.896211 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb"] Oct 30 21:31:41.898504 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:31:41.898476 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b8871bf_6ac4_4d11_a064_f0a762af556d.slice/crio-119fb624486561cfd81ed791811d5bf26e60044f71bff2dd216a2d27f859a244 WatchSource:0}: Error finding container 119fb624486561cfd81ed791811d5bf26e60044f71bff2dd216a2d27f859a244: Status 404 returned error can't find the container with id 119fb624486561cfd81ed791811d5bf26e60044f71bff2dd216a2d27f859a244 Oct 30 21:31:42.353960 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:42.353928 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" event={"ID":"5b8871bf-6ac4-4d11-a064-f0a762af556d","Type":"ContainerStarted","Data":"119fb624486561cfd81ed791811d5bf26e60044f71bff2dd216a2d27f859a244"} Oct 30 21:31:42.389895 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:42.389862 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:42.500844 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:42.500805 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:43.988879 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:43.988841 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:44.095842 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:44.095796 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:44.229982 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:44.229945 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:44.335098 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:44.335060 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:45.371948 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:45.371915 5801 generic.go:356] "Generic (PLEG): container finished" podID="5b8871bf-6ac4-4d11-a064-f0a762af556d" containerID="d66dceb73da444378fa11c274de7220807208e1f212fc87d93048f1785b13a8b" exitCode=0 Oct 30 21:31:45.372386 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:45.372002 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" event={"ID":"5b8871bf-6ac4-4d11-a064-f0a762af556d","Type":"ContainerDied","Data":"d66dceb73da444378fa11c274de7220807208e1f212fc87d93048f1785b13a8b"} Oct 30 21:31:46.378625 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:46.378589 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" event={"ID":"5b8871bf-6ac4-4d11-a064-f0a762af556d","Type":"ContainerStarted","Data":"a4f9552eb7b4d7081b0853da4def7e9a19f28a6b401f5c45f09581e51c2a278d"} Oct 30 21:31:46.379078 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:46.378746 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" Oct 30 21:31:46.429731 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:46.428872 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" podStartSLOduration=1.640589614 podStartE2EDuration="5.428856049s" totalImagesPullingTime="3.346306857s" totalInitContainerRuntime="441.959578ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:46.425663134 +0000 UTC m=+175.021479793" watchObservedRunningTime="2026-10-30 21:31:46.428856049 +0000 UTC m=+175.024672710" Oct 30 21:31:46.608799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:46.608767 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-mwmbn"] Oct 30 21:31:46.612114 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:46.612097 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mwmbn" Oct 30 21:31:46.616046 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:46.616016 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:31:46.616326 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:46.616303 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 21:31:46.621865 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:46.621840 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-mwmbn"] Oct 30 21:31:46.685529 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:46.685467 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9crtn\" (UniqueName: \"kubernetes.io/projected/296ecffb-f5b2-4152-8b81-d005552fe028-kube-api-access-9crtn\") pod \"jobset-operator-674cd59dc5-mwmbn\" (UID: \"296ecffb-f5b2-4152-8b81-d005552fe028\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mwmbn" Oct 30 21:31:46.685529 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:46.685509 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/296ecffb-f5b2-4152-8b81-d005552fe028-tmp\") pod \"jobset-operator-674cd59dc5-mwmbn\" (UID: \"296ecffb-f5b2-4152-8b81-d005552fe028\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mwmbn" Oct 30 21:31:46.701512 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:46.701494 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-6rbd9\"" Oct 30 21:31:46.786666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:46.786642 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9crtn\" (UniqueName: \"kubernetes.io/projected/296ecffb-f5b2-4152-8b81-d005552fe028-kube-api-access-9crtn\") pod \"jobset-operator-674cd59dc5-mwmbn\" (UID: \"296ecffb-f5b2-4152-8b81-d005552fe028\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mwmbn" Oct 30 21:31:46.786796 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:46.786677 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/296ecffb-f5b2-4152-8b81-d005552fe028-tmp\") pod \"jobset-operator-674cd59dc5-mwmbn\" (UID: \"296ecffb-f5b2-4152-8b81-d005552fe028\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mwmbn" Oct 30 21:31:46.786996 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:46.786979 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/296ecffb-f5b2-4152-8b81-d005552fe028-tmp\") pod \"jobset-operator-674cd59dc5-mwmbn\" (UID: \"296ecffb-f5b2-4152-8b81-d005552fe028\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mwmbn" Oct 30 21:31:46.799329 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:46.799300 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9crtn\" (UniqueName: \"kubernetes.io/projected/296ecffb-f5b2-4152-8b81-d005552fe028-kube-api-access-9crtn\") pod \"jobset-operator-674cd59dc5-mwmbn\" (UID: \"296ecffb-f5b2-4152-8b81-d005552fe028\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mwmbn" Oct 30 21:31:46.940938 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:46.940882 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mwmbn" Oct 30 21:31:47.214814 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:47.214787 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-mwmbn"] Oct 30 21:31:47.217249 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:31:47.217223 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod296ecffb_f5b2_4152_8b81_d005552fe028.slice/crio-13c647be386306b128c3e088503f0ae2849f3c79ca305db25451f0bf01f34815 WatchSource:0}: Error finding container 13c647be386306b128c3e088503f0ae2849f3c79ca305db25451f0bf01f34815: Status 404 returned error can't find the container with id 13c647be386306b128c3e088503f0ae2849f3c79ca305db25451f0bf01f34815 Oct 30 21:31:47.384389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:47.384351 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mwmbn" event={"ID":"296ecffb-f5b2-4152-8b81-d005552fe028","Type":"ContainerStarted","Data":"13c647be386306b128c3e088503f0ae2849f3c79ca305db25451f0bf01f34815"} Oct 30 21:31:48.153579 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:48.153538 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:50.399063 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:50.399029 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mwmbn" event={"ID":"296ecffb-f5b2-4152-8b81-d005552fe028","Type":"ContainerStarted","Data":"5935ef917519798518faeb730716d762a7210bc271bf93cef513044320b9af37"} Oct 30 21:31:50.418115 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:50.418069 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mwmbn" podStartSLOduration=2.278252789 podStartE2EDuration="4.418056981s" totalImagesPullingTime="2.139804192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:50.416897483 +0000 UTC m=+179.012714170" watchObservedRunningTime="2026-10-30 21:31:50.418056981 +0000 UTC m=+179.013873635" Oct 30 21:31:53.719549 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:53.719513 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:54.808518 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:54.808462 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:54.909729 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:54.909692 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:56.650385 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:56.650350 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:57.022822 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:57.022728 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:57.135078 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:57.135045 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:57.139391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:57.139365 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-sk2kw"] Oct 30 21:31:57.145774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:57.145741 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sk2kw" Oct 30 21:31:57.154434 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:57.154415 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 21:31:57.155451 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:57.155431 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 21:31:57.157001 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:57.156980 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-sk2kw"] Oct 30 21:31:57.264002 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:57.263976 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/faea70fa-2fa1-45b8-9409-72fb5d589cc3-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-sk2kw\" (UID: \"faea70fa-2fa1-45b8-9409-72fb5d589cc3\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sk2kw" Oct 30 21:31:57.264097 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:57.264034 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwj24\" (UniqueName: \"kubernetes.io/projected/faea70fa-2fa1-45b8-9409-72fb5d589cc3-kube-api-access-cwj24\") pod \"cert-manager-webhook-5dbbbf75cc-sk2kw\" (UID: \"faea70fa-2fa1-45b8-9409-72fb5d589cc3\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sk2kw" Oct 30 21:31:57.365192 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:57.365170 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwj24\" (UniqueName: \"kubernetes.io/projected/faea70fa-2fa1-45b8-9409-72fb5d589cc3-kube-api-access-cwj24\") pod \"cert-manager-webhook-5dbbbf75cc-sk2kw\" (UID: \"faea70fa-2fa1-45b8-9409-72fb5d589cc3\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sk2kw" Oct 30 21:31:57.365282 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:57.365213 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/faea70fa-2fa1-45b8-9409-72fb5d589cc3-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-sk2kw\" (UID: \"faea70fa-2fa1-45b8-9409-72fb5d589cc3\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sk2kw" Oct 30 21:31:57.379493 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:57.379472 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/faea70fa-2fa1-45b8-9409-72fb5d589cc3-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-sk2kw\" (UID: \"faea70fa-2fa1-45b8-9409-72fb5d589cc3\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sk2kw" Oct 30 21:31:57.379687 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:57.379666 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwj24\" (UniqueName: \"kubernetes.io/projected/faea70fa-2fa1-45b8-9409-72fb5d589cc3-kube-api-access-cwj24\") pod \"cert-manager-webhook-5dbbbf75cc-sk2kw\" (UID: \"faea70fa-2fa1-45b8-9409-72fb5d589cc3\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sk2kw" Oct 30 21:31:57.387264 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:57.387245 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bbc596d7f-nqfhb" Oct 30 21:31:57.456577 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:57.456553 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sk2kw" Oct 30 21:31:57.743250 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:57.743227 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-sk2kw"] Oct 30 21:31:57.745636 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:31:57.745608 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfaea70fa_2fa1_45b8_9409_72fb5d589cc3.slice/crio-dc518694916c933c2d98ed0cfbe8d6103a1b958de46f2b8c96568d9a2fd1f1f4 WatchSource:0}: Error finding container dc518694916c933c2d98ed0cfbe8d6103a1b958de46f2b8c96568d9a2fd1f1f4: Status 404 returned error can't find the container with id dc518694916c933c2d98ed0cfbe8d6103a1b958de46f2b8c96568d9a2fd1f1f4 Oct 30 21:31:57.959984 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:57.959951 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:58.434042 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:58.434011 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sk2kw" event={"ID":"faea70fa-2fa1-45b8-9409-72fb5d589cc3","Type":"ContainerStarted","Data":"dc518694916c933c2d98ed0cfbe8d6103a1b958de46f2b8c96568d9a2fd1f1f4"} Oct 30 21:31:59.385353 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:59.385314 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:59.606211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:31:59.606173 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:00.281067 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:32:00.281026 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:32:01.449691 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:01.449657 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sk2kw" event={"ID":"faea70fa-2fa1-45b8-9409-72fb5d589cc3","Type":"ContainerStarted","Data":"fe7d7b652140bf098306a39d1761b35b79849667a14594f972cca3b48987cd5c"} Oct 30 21:32:01.450201 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:01.449801 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sk2kw" Oct 30 21:32:01.475217 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:01.475173 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sk2kw" podStartSLOduration=1.655520401 podStartE2EDuration="4.47516106s" totalImagesPullingTime="2.819640659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:01.473049599 +0000 UTC m=+190.068866263" watchObservedRunningTime="2026-10-30 21:32:01.47516106 +0000 UTC m=+190.070977719" Oct 30 21:32:02.194955 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:02.194918 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:03.020496 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:03.020419 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:03.961369 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:03.961337 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:04.078714 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:04.078685 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-8rpvt"] Oct 30 21:32:04.101088 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:04.101065 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-8rpvt"] Oct 30 21:32:04.101216 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:04.101150 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8rpvt" Oct 30 21:32:04.105921 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:04.105899 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-qwkz9\"" Oct 30 21:32:04.122421 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:04.122399 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62ee9bc1-241a-4e58-8158-3e067cd0d10c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-8rpvt\" (UID: \"62ee9bc1-241a-4e58-8158-3e067cd0d10c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8rpvt" Oct 30 21:32:04.122527 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:04.122430 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmbtj\" (UniqueName: \"kubernetes.io/projected/62ee9bc1-241a-4e58-8158-3e067cd0d10c-kube-api-access-zmbtj\") pod \"cert-manager-cainjector-5c57bbd4fc-8rpvt\" (UID: \"62ee9bc1-241a-4e58-8158-3e067cd0d10c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8rpvt" Oct 30 21:32:04.223389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:04.223324 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62ee9bc1-241a-4e58-8158-3e067cd0d10c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-8rpvt\" (UID: \"62ee9bc1-241a-4e58-8158-3e067cd0d10c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8rpvt" Oct 30 21:32:04.223389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:04.223357 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmbtj\" (UniqueName: \"kubernetes.io/projected/62ee9bc1-241a-4e58-8158-3e067cd0d10c-kube-api-access-zmbtj\") pod \"cert-manager-cainjector-5c57bbd4fc-8rpvt\" (UID: \"62ee9bc1-241a-4e58-8158-3e067cd0d10c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8rpvt" Oct 30 21:32:04.235901 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:04.235872 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62ee9bc1-241a-4e58-8158-3e067cd0d10c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-8rpvt\" (UID: \"62ee9bc1-241a-4e58-8158-3e067cd0d10c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8rpvt" Oct 30 21:32:04.236026 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:04.236007 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmbtj\" (UniqueName: \"kubernetes.io/projected/62ee9bc1-241a-4e58-8158-3e067cd0d10c-kube-api-access-zmbtj\") pod \"cert-manager-cainjector-5c57bbd4fc-8rpvt\" (UID: \"62ee9bc1-241a-4e58-8158-3e067cd0d10c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8rpvt" Oct 30 21:32:04.432287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:04.432254 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8rpvt" Oct 30 21:32:04.573987 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:04.573954 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:04.673912 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:04.673879 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:04.701935 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:04.701867 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-8rpvt"] Oct 30 21:32:04.704398 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:32:04.704368 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62ee9bc1_241a_4e58_8158_3e067cd0d10c.slice/crio-aba637ef2054c7f74b2ff060466895035a26909d78fb738051ecf5f8a079843a WatchSource:0}: Error finding container aba637ef2054c7f74b2ff060466895035a26909d78fb738051ecf5f8a079843a: Status 404 returned error can't find the container with id aba637ef2054c7f74b2ff060466895035a26909d78fb738051ecf5f8a079843a Oct 30 21:32:05.468641 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:05.468607 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8rpvt" event={"ID":"62ee9bc1-241a-4e58-8158-3e067cd0d10c","Type":"ContainerStarted","Data":"827760dbf5f806b41af2c8a365da95a3e9418a4dbed978384a0d8c8fd6870f79"} Oct 30 21:32:05.469041 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:05.468648 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8rpvt" event={"ID":"62ee9bc1-241a-4e58-8158-3e067cd0d10c","Type":"ContainerStarted","Data":"aba637ef2054c7f74b2ff060466895035a26909d78fb738051ecf5f8a079843a"} Oct 30 21:32:05.489390 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:05.489345 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8rpvt" podStartSLOduration=1.489332259 podStartE2EDuration="1.489332259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:05.488245631 +0000 UTC m=+194.084062292" watchObservedRunningTime="2026-10-30 21:32:05.489332259 +0000 UTC m=+194.085148918" Oct 30 21:32:05.619334 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:05.619301 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:05.760495 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:05.760418 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:06.125771 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:06.125730 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:06.524504 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:06.524425 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:06.632061 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:06.632025 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:06.728152 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:06.728120 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:06.998546 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:06.998516 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:07.455970 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:07.455944 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sk2kw" Oct 30 21:32:08.709901 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:08.709870 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:08.836546 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:08.836505 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:09.973199 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:09.973152 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:12.548687 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:12.548655 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:13.158181 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:13.158146 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:13.523336 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:13.523206 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:13.650204 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:13.650167 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:13.954473 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:13.954437 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5978df7f68-prrl7"] Oct 30 21:32:13.958816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:13.958791 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5978df7f68-prrl7" Oct 30 21:32:13.969009 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:13.968986 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 21:32:13.969267 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:13.969075 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 21:32:13.969267 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:13.969115 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-65kvc\"" Oct 30 21:32:13.972131 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:13.972101 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5978df7f68-prrl7"] Oct 30 21:32:14.002670 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:14.002636 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56bc9daf-0942-46f1-aa51-49b8c868aa77-tmp\") pod \"kube-auth-proxy-5978df7f68-prrl7\" (UID: \"56bc9daf-0942-46f1-aa51-49b8c868aa77\") " pod="openshift-ingress/kube-auth-proxy-5978df7f68-prrl7" Oct 30 21:32:14.002841 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:14.002681 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6bsh\" (UniqueName: \"kubernetes.io/projected/56bc9daf-0942-46f1-aa51-49b8c868aa77-kube-api-access-b6bsh\") pod \"kube-auth-proxy-5978df7f68-prrl7\" (UID: \"56bc9daf-0942-46f1-aa51-49b8c868aa77\") " pod="openshift-ingress/kube-auth-proxy-5978df7f68-prrl7" Oct 30 21:32:14.002841 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:14.002760 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56bc9daf-0942-46f1-aa51-49b8c868aa77-tls-certs\") pod \"kube-auth-proxy-5978df7f68-prrl7\" (UID: \"56bc9daf-0942-46f1-aa51-49b8c868aa77\") " pod="openshift-ingress/kube-auth-proxy-5978df7f68-prrl7" Oct 30 21:32:14.033515 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:14.033466 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:14.103407 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:14.103368 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56bc9daf-0942-46f1-aa51-49b8c868aa77-tmp\") pod \"kube-auth-proxy-5978df7f68-prrl7\" (UID: \"56bc9daf-0942-46f1-aa51-49b8c868aa77\") " pod="openshift-ingress/kube-auth-proxy-5978df7f68-prrl7" Oct 30 21:32:14.103589 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:14.103416 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6bsh\" (UniqueName: \"kubernetes.io/projected/56bc9daf-0942-46f1-aa51-49b8c868aa77-kube-api-access-b6bsh\") pod \"kube-auth-proxy-5978df7f68-prrl7\" (UID: \"56bc9daf-0942-46f1-aa51-49b8c868aa77\") " pod="openshift-ingress/kube-auth-proxy-5978df7f68-prrl7" Oct 30 21:32:14.103589 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:14.103436 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56bc9daf-0942-46f1-aa51-49b8c868aa77-tls-certs\") pod \"kube-auth-proxy-5978df7f68-prrl7\" (UID: \"56bc9daf-0942-46f1-aa51-49b8c868aa77\") " pod="openshift-ingress/kube-auth-proxy-5978df7f68-prrl7" Oct 30 21:32:14.103711 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:32:14.103592 5801 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Oct 30 21:32:14.103711 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:32:14.103687 5801 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56bc9daf-0942-46f1-aa51-49b8c868aa77-tls-certs podName:56bc9daf-0942-46f1-aa51-49b8c868aa77 nodeName:}" failed. No retries permitted until 2026-10-30 21:32:14.60366826 +0000 UTC m=+203.199484919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/56bc9daf-0942-46f1-aa51-49b8c868aa77-tls-certs") pod "kube-auth-proxy-5978df7f68-prrl7" (UID: "56bc9daf-0942-46f1-aa51-49b8c868aa77") : secret "kube-auth-proxy-tls" not found Oct 30 21:32:14.105934 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:14.105908 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56bc9daf-0942-46f1-aa51-49b8c868aa77-tmp\") pod \"kube-auth-proxy-5978df7f68-prrl7\" (UID: \"56bc9daf-0942-46f1-aa51-49b8c868aa77\") " pod="openshift-ingress/kube-auth-proxy-5978df7f68-prrl7" Oct 30 21:32:14.135100 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:14.135062 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6bsh\" (UniqueName: \"kubernetes.io/projected/56bc9daf-0942-46f1-aa51-49b8c868aa77-kube-api-access-b6bsh\") pod \"kube-auth-proxy-5978df7f68-prrl7\" (UID: \"56bc9daf-0942-46f1-aa51-49b8c868aa77\") " pod="openshift-ingress/kube-auth-proxy-5978df7f68-prrl7" Oct 30 21:32:14.271860 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:14.271747 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:14.462942 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:14.462907 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:14.607276 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:14.607243 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56bc9daf-0942-46f1-aa51-49b8c868aa77-tls-certs\") pod \"kube-auth-proxy-5978df7f68-prrl7\" (UID: \"56bc9daf-0942-46f1-aa51-49b8c868aa77\") " pod="openshift-ingress/kube-auth-proxy-5978df7f68-prrl7" Oct 30 21:32:14.609850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:14.609822 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56bc9daf-0942-46f1-aa51-49b8c868aa77-tls-certs\") pod \"kube-auth-proxy-5978df7f68-prrl7\" (UID: \"56bc9daf-0942-46f1-aa51-49b8c868aa77\") " pod="openshift-ingress/kube-auth-proxy-5978df7f68-prrl7" Oct 30 21:32:14.875514 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:14.875430 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5978df7f68-prrl7" Oct 30 21:32:15.196354 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:15.196308 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5978df7f68-prrl7"] Oct 30 21:32:15.198795 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:32:15.198738 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56bc9daf_0942_46f1_aa51_49b8c868aa77.slice/crio-f96a80db9364bd91715942f0a2fc16e59ee7a378fb99e8cb39b6e0d1dc702cc1 WatchSource:0}: Error finding container f96a80db9364bd91715942f0a2fc16e59ee7a378fb99e8cb39b6e0d1dc702cc1: Status 404 returned error can't find the container with id f96a80db9364bd91715942f0a2fc16e59ee7a378fb99e8cb39b6e0d1dc702cc1 Oct 30 21:32:15.512506 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:15.512419 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5978df7f68-prrl7" event={"ID":"56bc9daf-0942-46f1-aa51-49b8c868aa77","Type":"ContainerStarted","Data":"f96a80db9364bd91715942f0a2fc16e59ee7a378fb99e8cb39b6e0d1dc702cc1"} Oct 30 21:32:15.860638 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:15.860608 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-fhf5g"] Oct 30 21:32:15.881023 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:15.880963 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-fhf5g" Oct 30 21:32:15.884886 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:15.884854 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-fhf5g"] Oct 30 21:32:15.904078 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:15.904053 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-kv9jg\"" Oct 30 21:32:15.906410 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:15.906371 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 21:32:15.906538 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:15.906468 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 21:32:15.917066 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:15.917034 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp555\" (UniqueName: \"kubernetes.io/projected/fc623bd2-899d-4cfb-ac2b-4f6a8d0bd00d-kube-api-access-lp555\") pod \"authorino-operator-7b7bb4bc77-fhf5g\" (UID: \"fc623bd2-899d-4cfb-ac2b-4f6a8d0bd00d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-fhf5g" Oct 30 21:32:15.918138 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:15.918110 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:16.018145 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:16.018106 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp555\" (UniqueName: \"kubernetes.io/projected/fc623bd2-899d-4cfb-ac2b-4f6a8d0bd00d-kube-api-access-lp555\") pod \"authorino-operator-7b7bb4bc77-fhf5g\" (UID: \"fc623bd2-899d-4cfb-ac2b-4f6a8d0bd00d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-fhf5g" Oct 30 21:32:16.051062 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:16.051033 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp555\" (UniqueName: \"kubernetes.io/projected/fc623bd2-899d-4cfb-ac2b-4f6a8d0bd00d-kube-api-access-lp555\") pod \"authorino-operator-7b7bb4bc77-fhf5g\" (UID: \"fc623bd2-899d-4cfb-ac2b-4f6a8d0bd00d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-fhf5g" Oct 30 21:32:16.216422 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:16.216333 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-fhf5g" Oct 30 21:32:16.249190 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:16.249137 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:16.505176 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:16.505093 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:16.519123 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:16.519089 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-fhf5g"] Oct 30 21:32:16.522612 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:32:16.522578 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc623bd2_899d_4cfb_ac2b_4f6a8d0bd00d.slice/crio-963133afe841bb73433b48f2826ff02f16f179fb8a3dced0362482cddb186fc8 WatchSource:0}: Error finding container 963133afe841bb73433b48f2826ff02f16f179fb8a3dced0362482cddb186fc8: Status 404 returned error can't find the container with id 963133afe841bb73433b48f2826ff02f16f179fb8a3dced0362482cddb186fc8 Oct 30 21:32:17.506027 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:17.505898 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:17.530648 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:17.530612 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-fhf5g" event={"ID":"fc623bd2-899d-4cfb-ac2b-4f6a8d0bd00d","Type":"ContainerStarted","Data":"963133afe841bb73433b48f2826ff02f16f179fb8a3dced0362482cddb186fc8"} Oct 30 21:32:17.687617 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:17.687580 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:18.537840 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:18.537796 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5978df7f68-prrl7" event={"ID":"56bc9daf-0942-46f1-aa51-49b8c868aa77","Type":"ContainerStarted","Data":"13855268b7718c18da528efd15fd9a2eec51dd566e792ccbce36bb4a8a410cd1"} Oct 30 21:32:18.592337 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:18.592284 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5978df7f68-prrl7" podStartSLOduration=2.8735352560000003 podStartE2EDuration="5.592265147s" totalImagesPullingTime="2.718729891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:18.589680099 +0000 UTC m=+207.185496788" watchObservedRunningTime="2026-10-30 21:32:18.592265147 +0000 UTC m=+207.188081810" Oct 30 21:32:19.543744 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:19.543645 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-fhf5g" event={"ID":"fc623bd2-899d-4cfb-ac2b-4f6a8d0bd00d","Type":"ContainerStarted","Data":"409edb863b3baa9f784f83ab5e3e07cdc25db68f61de60de1b930007b9e8cbd4"} Oct 30 21:32:19.544183 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:19.543784 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-fhf5g" Oct 30 21:32:19.569431 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:19.569369 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-fhf5g" podStartSLOduration=1.8682252350000002 podStartE2EDuration="4.569354882s" totalImagesPullingTime="2.701129647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:19.564197741 +0000 UTC m=+208.160014433" watchObservedRunningTime="2026-10-30 21:32:19.569354882 +0000 UTC m=+208.165171546" Oct 30 21:32:22.409408 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:22.409367 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:23.157267 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:23.157232 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:29.319740 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:29.319701 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:29.447677 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:29.447640 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:30.276512 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:32:30.276468 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:32:30.549728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:30.549656 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-fhf5g" Oct 30 21:32:31.189809 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:31.189772 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:31.372126 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:31.372089 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:31.748013 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:31.747976 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5c988cdb58-dr7v5"] Oct 30 21:32:31.752677 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:31.752657 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5c988cdb58-dr7v5" Oct 30 21:32:31.758769 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:31.758736 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 21:32:31.770745 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:31.770722 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 21:32:31.770745 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:31.770777 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-49l6l\"" Oct 30 21:32:31.773454 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:31.773424 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5c988cdb58-dr7v5"] Oct 30 21:32:31.777071 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:31.777055 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 21:32:31.849608 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:31.849582 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388-cert\") pod \"jobset-controller-manager-5c988cdb58-dr7v5\" (UID: \"b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388\") " pod="openshift-jobset-operator/jobset-controller-manager-5c988cdb58-dr7v5" Oct 30 21:32:31.849777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:31.849630 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388-manager-config\") pod \"jobset-controller-manager-5c988cdb58-dr7v5\" (UID: \"b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388\") " pod="openshift-jobset-operator/jobset-controller-manager-5c988cdb58-dr7v5" Oct 30 21:32:31.849777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:31.849742 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sjdz\" (UniqueName: \"kubernetes.io/projected/b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388-kube-api-access-6sjdz\") pod \"jobset-controller-manager-5c988cdb58-dr7v5\" (UID: \"b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388\") " pod="openshift-jobset-operator/jobset-controller-manager-5c988cdb58-dr7v5" Oct 30 21:32:31.849872 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:31.849796 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388-metrics-certs\") pod \"jobset-controller-manager-5c988cdb58-dr7v5\" (UID: \"b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388\") " pod="openshift-jobset-operator/jobset-controller-manager-5c988cdb58-dr7v5" Oct 30 21:32:31.950426 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:31.950392 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388-manager-config\") pod \"jobset-controller-manager-5c988cdb58-dr7v5\" (UID: \"b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388\") " pod="openshift-jobset-operator/jobset-controller-manager-5c988cdb58-dr7v5" Oct 30 21:32:31.950564 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:31.950449 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sjdz\" (UniqueName: \"kubernetes.io/projected/b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388-kube-api-access-6sjdz\") pod \"jobset-controller-manager-5c988cdb58-dr7v5\" (UID: \"b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388\") " pod="openshift-jobset-operator/jobset-controller-manager-5c988cdb58-dr7v5" Oct 30 21:32:31.950564 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:31.950489 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388-metrics-certs\") pod \"jobset-controller-manager-5c988cdb58-dr7v5\" (UID: \"b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388\") " pod="openshift-jobset-operator/jobset-controller-manager-5c988cdb58-dr7v5" Oct 30 21:32:31.950564 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:31.950536 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388-cert\") pod \"jobset-controller-manager-5c988cdb58-dr7v5\" (UID: \"b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388\") " pod="openshift-jobset-operator/jobset-controller-manager-5c988cdb58-dr7v5" Oct 30 21:32:31.951216 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:31.951195 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388-manager-config\") pod \"jobset-controller-manager-5c988cdb58-dr7v5\" (UID: \"b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388\") " pod="openshift-jobset-operator/jobset-controller-manager-5c988cdb58-dr7v5" Oct 30 21:32:31.953116 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:31.953093 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388-metrics-certs\") pod \"jobset-controller-manager-5c988cdb58-dr7v5\" (UID: \"b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388\") " pod="openshift-jobset-operator/jobset-controller-manager-5c988cdb58-dr7v5" Oct 30 21:32:31.953196 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:31.953152 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388-cert\") pod \"jobset-controller-manager-5c988cdb58-dr7v5\" (UID: \"b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388\") " pod="openshift-jobset-operator/jobset-controller-manager-5c988cdb58-dr7v5" Oct 30 21:32:31.974631 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:31.974610 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sjdz\" (UniqueName: \"kubernetes.io/projected/b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388-kube-api-access-6sjdz\") pod \"jobset-controller-manager-5c988cdb58-dr7v5\" (UID: \"b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388\") " pod="openshift-jobset-operator/jobset-controller-manager-5c988cdb58-dr7v5" Oct 30 21:32:32.064295 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:32.064266 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5c988cdb58-dr7v5" Oct 30 21:32:32.357527 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:32.357505 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5c988cdb58-dr7v5"] Oct 30 21:32:32.359571 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:32:32.359543 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ba8f2e_a8c3_4cdc_8cb4_9716a4c6a388.slice/crio-39b5326a2561169567f0743eaf8c3035fac27036919f1bffa4929eb4303ba6a0 WatchSource:0}: Error finding container 39b5326a2561169567f0743eaf8c3035fac27036919f1bffa4929eb4303ba6a0: Status 404 returned error can't find the container with id 39b5326a2561169567f0743eaf8c3035fac27036919f1bffa4929eb4303ba6a0 Oct 30 21:32:32.602651 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:32.602613 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5c988cdb58-dr7v5" event={"ID":"b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388","Type":"ContainerStarted","Data":"39b5326a2561169567f0743eaf8c3035fac27036919f1bffa4929eb4303ba6a0"} Oct 30 21:32:34.052885 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:34.052849 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:34.191238 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:34.191206 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:35.619063 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:35.619031 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5c988cdb58-dr7v5" event={"ID":"b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388","Type":"ContainerStarted","Data":"40be6d35093be0199a050813cca60642a94d87cdc0a3912b0d1f96f32335cb29"} Oct 30 21:32:35.619451 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:35.619133 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5c988cdb58-dr7v5" Oct 30 21:32:35.645058 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:35.645011 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5c988cdb58-dr7v5" podStartSLOduration=1.790542931 podStartE2EDuration="4.644998492s" totalImagesPullingTime="2.854455561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:35.642656499 +0000 UTC m=+224.238473183" watchObservedRunningTime="2026-10-30 21:32:35.644998492 +0000 UTC m=+224.240815151" Oct 30 21:32:36.639841 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:36.639806 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:36.760893 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:36.760859 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:41.347323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:41.347284 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:41.487544 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:41.487508 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:41.616053 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:41.615959 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:46.628820 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:46.628789 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5c988cdb58-dr7v5" Oct 30 21:32:48.401521 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:48.401489 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:48.522342 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:48.522311 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:49.880592 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:49.880555 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:53.328980 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:53.328945 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:53.493163 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:53.493123 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:55.784052 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:55.784009 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:55.891049 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:55.891013 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:56.707893 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:56.707851 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:59.009458 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:59.009414 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:59.129133 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:59.129099 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:59.230276 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:59.230246 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:59.917465 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:32:59.917429 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:00.019829 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:00.019792 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:00.277483 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:33:00.277399 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:33:01.157716 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:01.157677 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:01.262171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:01.262136 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:01.560210 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:01.560168 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:01.671102 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:01.671067 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:01.997642 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:01.997559 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:02.113158 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:02.113120 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:08.258182 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:08.258145 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:08.363790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:08.363736 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:09.420677 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:09.420644 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:09.600300 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:09.600264 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:13.018389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:13.018301 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:13.189763 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:13.189718 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:15.504813 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:15.504775 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:15.733203 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:15.733175 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:15.923010 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:15.922971 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:16.221797 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:16.221689 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:17.905019 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:17.904983 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:18.009811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:18.009777 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:18.120250 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:18.120215 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:18.229416 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:18.229339 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:18.671700 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:18.671670 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:30.277318 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:33:30.277281 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:33:39.304584 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:39.304498 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:39.413247 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:39.413207 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:39.516412 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:39.516357 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:39.617588 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:39.617493 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:43.259719 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:43.259670 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:43.367508 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:43.367464 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:45.021741 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:45.021701 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:45.178977 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:45.178948 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:56.770223 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:56.770188 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:59.017928 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:33:59.017889 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:00.276380 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:34:00.276345 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:34:15.360218 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:15.360180 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:15.650865 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:15.650744 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:16.843674 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:16.843637 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:17.068678 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:17.068644 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:26.796967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:26.796927 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:26.897047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:26.897005 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:27.124473 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:27.124405 5801 generic.go:356] "Generic (PLEG): container finished" podID="79d7a2c8-0585-48a3-88d4-d49b23672216" containerID="63338caca66d5d40f5435f9815a1df944f421544469b85e1be47e43bd2b3733e" exitCode=0 Oct 30 21:34:27.124609 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:27.124484 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" event={"ID":"79d7a2c8-0585-48a3-88d4-d49b23672216","Type":"ContainerDied","Data":"63338caca66d5d40f5435f9815a1df944f421544469b85e1be47e43bd2b3733e"} Oct 30 21:34:27.124838 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:27.124825 5801 scope.go:122] "RemoveContainer" containerID="63338caca66d5d40f5435f9815a1df944f421544469b85e1be47e43bd2b3733e" Oct 30 21:34:27.125929 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:27.125904 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:34:27.125994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:27.125969 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:34:27.126037 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:27.125996 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:34:28.130665 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:28.130629 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" event={"ID":"79d7a2c8-0585-48a3-88d4-d49b23672216","Type":"ContainerStarted","Data":"8eeff9e295780e2ce15ac0f0d42f372d5844b94d28b2a1d5bdc026100dad48d5"} Oct 30 21:34:28.769777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:28.768327 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:34:28.773881 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:28.773850 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-448hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Oct 30 21:34:28.774117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:28.774087 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" podUID="79d7a2c8-0585-48a3-88d4-d49b23672216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Oct 30 21:34:29.773775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:29.768207 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-448hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Oct 30 21:34:29.773775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:29.768276 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" podUID="79d7a2c8-0585-48a3-88d4-d49b23672216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Oct 30 21:34:30.300138 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:34:30.300103 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:34:30.769520 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:30.769089 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-448hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Oct 30 21:34:30.769520 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:30.769149 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" podUID="79d7a2c8-0585-48a3-88d4-d49b23672216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Oct 30 21:34:31.769688 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:31.769651 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-448hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Oct 30 21:34:31.770295 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:31.770266 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" podUID="79d7a2c8-0585-48a3-88d4-d49b23672216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Oct 30 21:34:32.143711 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:32.143679 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:34:32.767778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:32.764130 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-448hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Oct 30 21:34:32.767778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:32.764198 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" podUID="79d7a2c8-0585-48a3-88d4-d49b23672216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Oct 30 21:34:33.767569 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:33.767535 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:34:33.768516 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:33.768496 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 21:34:56.820459 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:34:56.820422 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:00.276250 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:35:00.276216 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:35:02.987732 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:02.987621 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:03.089980 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:03.089953 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:04.419209 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:04.419177 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:04.519696 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:04.519665 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:05.327617 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:05.327581 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:05.427795 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:05.427740 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:22.930297 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:22.930245 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:23.986692 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:23.986656 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf"] Oct 30 21:35:23.990507 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:23.990487 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" Oct 30 21:35:23.995101 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:23.995077 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 21:35:23.996345 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:23.996319 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 21:35:23.996456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:23.996438 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-k8tnn\"" Oct 30 21:35:23.997902 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:23.997885 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 21:35:24.000292 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:24.000272 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf"] Oct 30 21:35:24.069358 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:24.069324 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e0434b3-6ab0-42c5-9c5b-e4592daadca7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6vklf\" (UID: \"7e0434b3-6ab0-42c5-9c5b-e4592daadca7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" Oct 30 21:35:24.069536 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:24.069362 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e0434b3-6ab0-42c5-9c5b-e4592daadca7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6vklf\" (UID: \"7e0434b3-6ab0-42c5-9c5b-e4592daadca7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" Oct 30 21:35:24.069536 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:24.069398 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e0434b3-6ab0-42c5-9c5b-e4592daadca7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6vklf\" (UID: \"7e0434b3-6ab0-42c5-9c5b-e4592daadca7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" Oct 30 21:35:24.069619 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:24.069534 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e0434b3-6ab0-42c5-9c5b-e4592daadca7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6vklf\" (UID: \"7e0434b3-6ab0-42c5-9c5b-e4592daadca7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" Oct 30 21:35:24.069619 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:24.069560 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e0434b3-6ab0-42c5-9c5b-e4592daadca7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6vklf\" (UID: \"7e0434b3-6ab0-42c5-9c5b-e4592daadca7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" Oct 30 21:35:24.069619 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:24.069596 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86g5s\" (UniqueName: \"kubernetes.io/projected/7e0434b3-6ab0-42c5-9c5b-e4592daadca7-kube-api-access-86g5s\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6vklf\" (UID: \"7e0434b3-6ab0-42c5-9c5b-e4592daadca7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" Oct 30 21:35:24.075380 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:24.075351 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:24.170552 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:24.170509 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e0434b3-6ab0-42c5-9c5b-e4592daadca7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6vklf\" (UID: \"7e0434b3-6ab0-42c5-9c5b-e4592daadca7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" Oct 30 21:35:24.170552 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:24.170549 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e0434b3-6ab0-42c5-9c5b-e4592daadca7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6vklf\" (UID: \"7e0434b3-6ab0-42c5-9c5b-e4592daadca7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" Oct 30 21:35:24.170786 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:24.170678 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86g5s\" (UniqueName: \"kubernetes.io/projected/7e0434b3-6ab0-42c5-9c5b-e4592daadca7-kube-api-access-86g5s\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6vklf\" (UID: \"7e0434b3-6ab0-42c5-9c5b-e4592daadca7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" Oct 30 21:35:24.170786 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:24.170736 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e0434b3-6ab0-42c5-9c5b-e4592daadca7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6vklf\" (UID: \"7e0434b3-6ab0-42c5-9c5b-e4592daadca7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" Oct 30 21:35:24.170905 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:24.170792 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e0434b3-6ab0-42c5-9c5b-e4592daadca7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6vklf\" (UID: \"7e0434b3-6ab0-42c5-9c5b-e4592daadca7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" Oct 30 21:35:24.170905 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:24.170858 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e0434b3-6ab0-42c5-9c5b-e4592daadca7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6vklf\" (UID: \"7e0434b3-6ab0-42c5-9c5b-e4592daadca7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" Oct 30 21:35:24.171017 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:24.171006 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e0434b3-6ab0-42c5-9c5b-e4592daadca7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6vklf\" (UID: \"7e0434b3-6ab0-42c5-9c5b-e4592daadca7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" Oct 30 21:35:24.171240 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:24.171215 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e0434b3-6ab0-42c5-9c5b-e4592daadca7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6vklf\" (UID: \"7e0434b3-6ab0-42c5-9c5b-e4592daadca7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" Oct 30 21:35:24.171337 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:24.171276 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e0434b3-6ab0-42c5-9c5b-e4592daadca7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6vklf\" (UID: \"7e0434b3-6ab0-42c5-9c5b-e4592daadca7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" Oct 30 21:35:24.172903 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:24.172884 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e0434b3-6ab0-42c5-9c5b-e4592daadca7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6vklf\" (UID: \"7e0434b3-6ab0-42c5-9c5b-e4592daadca7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" Oct 30 21:35:24.173081 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:24.173063 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e0434b3-6ab0-42c5-9c5b-e4592daadca7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6vklf\" (UID: \"7e0434b3-6ab0-42c5-9c5b-e4592daadca7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" Oct 30 21:35:24.183037 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:24.183012 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86g5s\" (UniqueName: \"kubernetes.io/projected/7e0434b3-6ab0-42c5-9c5b-e4592daadca7-kube-api-access-86g5s\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6vklf\" (UID: \"7e0434b3-6ab0-42c5-9c5b-e4592daadca7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" Oct 30 21:35:24.303860 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:24.303835 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" Oct 30 21:35:24.578937 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:24.578865 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf"] Oct 30 21:35:24.581894 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:35:24.581866 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0434b3_6ab0_42c5_9c5b_e4592daadca7.slice/crio-4cd7bbb3ee22abd4b5e271d91a3fe943443f756f3b6324e4ff777e3d3188acc2 WatchSource:0}: Error finding container 4cd7bbb3ee22abd4b5e271d91a3fe943443f756f3b6324e4ff777e3d3188acc2: Status 404 returned error can't find the container with id 4cd7bbb3ee22abd4b5e271d91a3fe943443f756f3b6324e4ff777e3d3188acc2 Oct 30 21:35:24.841790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:24.841666 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:25.383889 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:25.383850 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" event={"ID":"7e0434b3-6ab0-42c5-9c5b-e4592daadca7","Type":"ContainerStarted","Data":"4cd7bbb3ee22abd4b5e271d91a3fe943443f756f3b6324e4ff777e3d3188acc2"} Oct 30 21:35:26.391533 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.391493 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" event={"ID":"7e0434b3-6ab0-42c5-9c5b-e4592daadca7","Type":"ContainerStarted","Data":"6e0c0eca9166280a8a45fd8f4db258a828b1720120b80f8d294ca093a196f7cb"} Oct 30 21:35:26.413987 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.413941 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" podStartSLOduration=2.15004808 podStartE2EDuration="3.413925399s" totalImagesPullingTime="1.263877319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:26.411477527 +0000 UTC m=+395.007294189" watchObservedRunningTime="2026-10-30 21:35:26.413925399 +0000 UTC m=+395.009742059" Oct 30 21:35:26.680851 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.680743 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv"] Oct 30 21:35:26.684215 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.684196 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" Oct 30 21:35:26.688940 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.688914 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 21:35:26.694035 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.694011 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv"] Oct 30 21:35:26.732098 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.732074 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:26.794013 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.793985 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dff9\" (UniqueName: \"kubernetes.io/projected/b6307469-5f7d-41d9-9490-e8381ff9bbfb-kube-api-access-7dff9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv\" (UID: \"b6307469-5f7d-41d9-9490-e8381ff9bbfb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" Oct 30 21:35:26.794163 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.794020 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6307469-5f7d-41d9-9490-e8381ff9bbfb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv\" (UID: \"b6307469-5f7d-41d9-9490-e8381ff9bbfb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" Oct 30 21:35:26.794163 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.794096 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6307469-5f7d-41d9-9490-e8381ff9bbfb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv\" (UID: \"b6307469-5f7d-41d9-9490-e8381ff9bbfb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" Oct 30 21:35:26.794230 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.794193 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6307469-5f7d-41d9-9490-e8381ff9bbfb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv\" (UID: \"b6307469-5f7d-41d9-9490-e8381ff9bbfb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" Oct 30 21:35:26.794268 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.794240 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6307469-5f7d-41d9-9490-e8381ff9bbfb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv\" (UID: \"b6307469-5f7d-41d9-9490-e8381ff9bbfb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" Oct 30 21:35:26.794301 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.794294 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6307469-5f7d-41d9-9490-e8381ff9bbfb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv\" (UID: \"b6307469-5f7d-41d9-9490-e8381ff9bbfb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" Oct 30 21:35:26.895488 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.895446 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6307469-5f7d-41d9-9490-e8381ff9bbfb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv\" (UID: \"b6307469-5f7d-41d9-9490-e8381ff9bbfb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" Oct 30 21:35:26.895488 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.895489 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dff9\" (UniqueName: \"kubernetes.io/projected/b6307469-5f7d-41d9-9490-e8381ff9bbfb-kube-api-access-7dff9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv\" (UID: \"b6307469-5f7d-41d9-9490-e8381ff9bbfb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" Oct 30 21:35:26.895767 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.895512 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6307469-5f7d-41d9-9490-e8381ff9bbfb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv\" (UID: \"b6307469-5f7d-41d9-9490-e8381ff9bbfb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" Oct 30 21:35:26.895767 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.895550 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6307469-5f7d-41d9-9490-e8381ff9bbfb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv\" (UID: \"b6307469-5f7d-41d9-9490-e8381ff9bbfb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" Oct 30 21:35:26.895767 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.895579 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6307469-5f7d-41d9-9490-e8381ff9bbfb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv\" (UID: \"b6307469-5f7d-41d9-9490-e8381ff9bbfb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" Oct 30 21:35:26.895767 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.895713 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6307469-5f7d-41d9-9490-e8381ff9bbfb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv\" (UID: \"b6307469-5f7d-41d9-9490-e8381ff9bbfb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" Oct 30 21:35:26.896035 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.896010 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6307469-5f7d-41d9-9490-e8381ff9bbfb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv\" (UID: \"b6307469-5f7d-41d9-9490-e8381ff9bbfb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" Oct 30 21:35:26.896147 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.896124 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6307469-5f7d-41d9-9490-e8381ff9bbfb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv\" (UID: \"b6307469-5f7d-41d9-9490-e8381ff9bbfb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" Oct 30 21:35:26.896197 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.896146 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6307469-5f7d-41d9-9490-e8381ff9bbfb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv\" (UID: \"b6307469-5f7d-41d9-9490-e8381ff9bbfb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" Oct 30 21:35:26.897977 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.897958 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6307469-5f7d-41d9-9490-e8381ff9bbfb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv\" (UID: \"b6307469-5f7d-41d9-9490-e8381ff9bbfb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" Oct 30 21:35:26.898056 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.898040 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6307469-5f7d-41d9-9490-e8381ff9bbfb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv\" (UID: \"b6307469-5f7d-41d9-9490-e8381ff9bbfb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" Oct 30 21:35:26.907736 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.907708 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dff9\" (UniqueName: \"kubernetes.io/projected/b6307469-5f7d-41d9-9490-e8381ff9bbfb-kube-api-access-7dff9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv\" (UID: \"b6307469-5f7d-41d9-9490-e8381ff9bbfb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" Oct 30 21:35:26.999944 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:26.999848 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" Oct 30 21:35:27.274607 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:27.274584 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv"] Oct 30 21:35:27.277126 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:35:27.277098 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6307469_5f7d_41d9_9490_e8381ff9bbfb.slice/crio-0ea1c206c5c65baa4a91fba2a2b9c7b8361befba10b10a058f0aedc510759d9a WatchSource:0}: Error finding container 0ea1c206c5c65baa4a91fba2a2b9c7b8361befba10b10a058f0aedc510759d9a: Status 404 returned error can't find the container with id 0ea1c206c5c65baa4a91fba2a2b9c7b8361befba10b10a058f0aedc510759d9a Oct 30 21:35:27.304161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:27.304137 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" Oct 30 21:35:27.397493 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:27.397457 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" event={"ID":"b6307469-5f7d-41d9-9490-e8381ff9bbfb","Type":"ContainerStarted","Data":"0ea1c206c5c65baa4a91fba2a2b9c7b8361befba10b10a058f0aedc510759d9a"} Oct 30 21:35:28.405682 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:28.405634 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" event={"ID":"b6307469-5f7d-41d9-9490-e8381ff9bbfb","Type":"ContainerStarted","Data":"907b266dd4d0e58e0143b09ae398ea83ebbb0eaf80804e2fb13035b28df1ef3e"} Oct 30 21:35:28.427512 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:28.427464 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" podStartSLOduration=2.091077016 podStartE2EDuration="2.427451096s" totalImagesPullingTime="336.37408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:28.426218447 +0000 UTC m=+397.022035128" watchObservedRunningTime="2026-10-30 21:35:28.427451096 +0000 UTC m=+397.023267756" Oct 30 21:35:28.704865 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:28.704742 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:29.000983 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:29.000906 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" Oct 30 21:35:30.137254 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.137223 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h"] Oct 30 21:35:30.140923 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.140900 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" Oct 30 21:35:30.155304 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.155277 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h"] Oct 30 21:35:30.157913 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.157892 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 21:35:30.193304 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.193257 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:30.226359 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.226324 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edecde9f-7b29-4563-bf29-bcb748233470-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h\" (UID: \"edecde9f-7b29-4563-bf29-bcb748233470\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" Oct 30 21:35:30.226502 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.226380 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edecde9f-7b29-4563-bf29-bcb748233470-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h\" (UID: \"edecde9f-7b29-4563-bf29-bcb748233470\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" Oct 30 21:35:30.226502 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.226444 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edecde9f-7b29-4563-bf29-bcb748233470-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h\" (UID: \"edecde9f-7b29-4563-bf29-bcb748233470\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" Oct 30 21:35:30.226586 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.226528 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edecde9f-7b29-4563-bf29-bcb748233470-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h\" (UID: \"edecde9f-7b29-4563-bf29-bcb748233470\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" Oct 30 21:35:30.226586 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.226578 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edecde9f-7b29-4563-bf29-bcb748233470-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h\" (UID: \"edecde9f-7b29-4563-bf29-bcb748233470\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" Oct 30 21:35:30.226650 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.226605 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ftt7\" (UniqueName: \"kubernetes.io/projected/edecde9f-7b29-4563-bf29-bcb748233470-kube-api-access-4ftt7\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h\" (UID: \"edecde9f-7b29-4563-bf29-bcb748233470\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" Oct 30 21:35:30.285829 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:35:30.285786 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:35:30.327805 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.327775 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edecde9f-7b29-4563-bf29-bcb748233470-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h\" (UID: \"edecde9f-7b29-4563-bf29-bcb748233470\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" Oct 30 21:35:30.327961 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.327834 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edecde9f-7b29-4563-bf29-bcb748233470-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h\" (UID: \"edecde9f-7b29-4563-bf29-bcb748233470\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" Oct 30 21:35:30.327961 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.327859 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edecde9f-7b29-4563-bf29-bcb748233470-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h\" (UID: \"edecde9f-7b29-4563-bf29-bcb748233470\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" Oct 30 21:35:30.327961 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.327885 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edecde9f-7b29-4563-bf29-bcb748233470-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h\" (UID: \"edecde9f-7b29-4563-bf29-bcb748233470\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" Oct 30 21:35:30.327961 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.327914 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edecde9f-7b29-4563-bf29-bcb748233470-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h\" (UID: \"edecde9f-7b29-4563-bf29-bcb748233470\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" Oct 30 21:35:30.327961 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.327947 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ftt7\" (UniqueName: \"kubernetes.io/projected/edecde9f-7b29-4563-bf29-bcb748233470-kube-api-access-4ftt7\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h\" (UID: \"edecde9f-7b29-4563-bf29-bcb748233470\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" Oct 30 21:35:30.328306 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.328279 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edecde9f-7b29-4563-bf29-bcb748233470-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h\" (UID: \"edecde9f-7b29-4563-bf29-bcb748233470\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" Oct 30 21:35:30.328363 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.328313 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edecde9f-7b29-4563-bf29-bcb748233470-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h\" (UID: \"edecde9f-7b29-4563-bf29-bcb748233470\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" Oct 30 21:35:30.328398 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.328381 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edecde9f-7b29-4563-bf29-bcb748233470-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h\" (UID: \"edecde9f-7b29-4563-bf29-bcb748233470\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" Oct 30 21:35:30.330104 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.330081 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edecde9f-7b29-4563-bf29-bcb748233470-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h\" (UID: \"edecde9f-7b29-4563-bf29-bcb748233470\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" Oct 30 21:35:30.330336 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.330312 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edecde9f-7b29-4563-bf29-bcb748233470-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h\" (UID: \"edecde9f-7b29-4563-bf29-bcb748233470\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" Oct 30 21:35:30.341221 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.341195 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ftt7\" (UniqueName: \"kubernetes.io/projected/edecde9f-7b29-4563-bf29-bcb748233470-kube-api-access-4ftt7\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h\" (UID: \"edecde9f-7b29-4563-bf29-bcb748233470\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" Oct 30 21:35:30.453225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.453138 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" Oct 30 21:35:30.763439 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:30.763363 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h"] Oct 30 21:35:30.767308 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:35:30.767278 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedecde9f_7b29_4563_bf29_bcb748233470.slice/crio-0869f4e6e290d5cfa304db6cfb82bfbd37f49172a9d8e54bbf6ae8b3e1d87a38 WatchSource:0}: Error finding container 0869f4e6e290d5cfa304db6cfb82bfbd37f49172a9d8e54bbf6ae8b3e1d87a38: Status 404 returned error can't find the container with id 0869f4e6e290d5cfa304db6cfb82bfbd37f49172a9d8e54bbf6ae8b3e1d87a38 Oct 30 21:35:31.286916 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.286885 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6"] Oct 30 21:35:31.290361 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.290335 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:35:31.295182 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.295162 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 21:35:31.304176 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.304154 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6"] Oct 30 21:35:31.346461 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.346429 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:31.421647 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.421564 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" event={"ID":"edecde9f-7b29-4563-bf29-bcb748233470","Type":"ContainerStarted","Data":"562ae6d20a6fac054022ee34af1087033c31dee359cdcf9289f7a9a12f368100"} Oct 30 21:35:31.421647 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.421622 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" event={"ID":"edecde9f-7b29-4563-bf29-bcb748233470","Type":"ContainerStarted","Data":"0869f4e6e290d5cfa304db6cfb82bfbd37f49172a9d8e54bbf6ae8b3e1d87a38"} Oct 30 21:35:31.437368 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.437337 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8c67e3a-dafa-4ed4-b5e3-f1b341427687-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6\" (UID: \"a8c67e3a-dafa-4ed4-b5e3-f1b341427687\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:35:31.437482 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.437404 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8c67e3a-dafa-4ed4-b5e3-f1b341427687-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6\" (UID: \"a8c67e3a-dafa-4ed4-b5e3-f1b341427687\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:35:31.437482 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.437457 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8c67e3a-dafa-4ed4-b5e3-f1b341427687-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6\" (UID: \"a8c67e3a-dafa-4ed4-b5e3-f1b341427687\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:35:31.437566 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.437550 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhgtj\" (UniqueName: \"kubernetes.io/projected/a8c67e3a-dafa-4ed4-b5e3-f1b341427687-kube-api-access-mhgtj\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6\" (UID: \"a8c67e3a-dafa-4ed4-b5e3-f1b341427687\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:35:31.437607 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.437584 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8c67e3a-dafa-4ed4-b5e3-f1b341427687-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6\" (UID: \"a8c67e3a-dafa-4ed4-b5e3-f1b341427687\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:35:31.437669 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.437656 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8c67e3a-dafa-4ed4-b5e3-f1b341427687-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6\" (UID: \"a8c67e3a-dafa-4ed4-b5e3-f1b341427687\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:35:31.445586 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.445541 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" podStartSLOduration=1.113482624 podStartE2EDuration="1.445530571s" totalImagesPullingTime="332.047947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:31.443777002 +0000 UTC m=+400.039593660" watchObservedRunningTime="2026-10-30 21:35:31.445530571 +0000 UTC m=+400.041347231" Oct 30 21:35:31.453590 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.453573 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" Oct 30 21:35:31.538240 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.538206 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8c67e3a-dafa-4ed4-b5e3-f1b341427687-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6\" (UID: \"a8c67e3a-dafa-4ed4-b5e3-f1b341427687\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:35:31.538441 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.538272 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhgtj\" (UniqueName: \"kubernetes.io/projected/a8c67e3a-dafa-4ed4-b5e3-f1b341427687-kube-api-access-mhgtj\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6\" (UID: \"a8c67e3a-dafa-4ed4-b5e3-f1b341427687\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:35:31.538441 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.538407 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8c67e3a-dafa-4ed4-b5e3-f1b341427687-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6\" (UID: \"a8c67e3a-dafa-4ed4-b5e3-f1b341427687\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:35:31.539092 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.538617 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8c67e3a-dafa-4ed4-b5e3-f1b341427687-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6\" (UID: \"a8c67e3a-dafa-4ed4-b5e3-f1b341427687\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:35:31.539092 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.538695 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8c67e3a-dafa-4ed4-b5e3-f1b341427687-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6\" (UID: \"a8c67e3a-dafa-4ed4-b5e3-f1b341427687\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:35:31.539092 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.538732 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8c67e3a-dafa-4ed4-b5e3-f1b341427687-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6\" (UID: \"a8c67e3a-dafa-4ed4-b5e3-f1b341427687\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:35:31.539488 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.539310 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8c67e3a-dafa-4ed4-b5e3-f1b341427687-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6\" (UID: \"a8c67e3a-dafa-4ed4-b5e3-f1b341427687\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:35:31.539488 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.539342 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8c67e3a-dafa-4ed4-b5e3-f1b341427687-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6\" (UID: \"a8c67e3a-dafa-4ed4-b5e3-f1b341427687\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:35:31.539593 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.539556 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8c67e3a-dafa-4ed4-b5e3-f1b341427687-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6\" (UID: \"a8c67e3a-dafa-4ed4-b5e3-f1b341427687\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:35:31.541148 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.541123 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8c67e3a-dafa-4ed4-b5e3-f1b341427687-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6\" (UID: \"a8c67e3a-dafa-4ed4-b5e3-f1b341427687\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:35:31.541412 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.541392 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8c67e3a-dafa-4ed4-b5e3-f1b341427687-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6\" (UID: \"a8c67e3a-dafa-4ed4-b5e3-f1b341427687\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:35:31.553189 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.553170 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhgtj\" (UniqueName: \"kubernetes.io/projected/a8c67e3a-dafa-4ed4-b5e3-f1b341427687-kube-api-access-mhgtj\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6\" (UID: \"a8c67e3a-dafa-4ed4-b5e3-f1b341427687\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:35:31.604327 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.604293 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:35:31.877944 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:31.877917 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6"] Oct 30 21:35:31.880154 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:35:31.880127 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8c67e3a_dafa_4ed4_b5e3_f1b341427687.slice/crio-eccb01b1d39420aa8960aafbe0dda28508606acff74dc191d13627699ca5da4c WatchSource:0}: Error finding container eccb01b1d39420aa8960aafbe0dda28508606acff74dc191d13627699ca5da4c: Status 404 returned error can't find the container with id eccb01b1d39420aa8960aafbe0dda28508606acff74dc191d13627699ca5da4c Oct 30 21:35:32.427803 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:32.427741 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" event={"ID":"a8c67e3a-dafa-4ed4-b5e3-f1b341427687","Type":"ContainerStarted","Data":"692914bf525ce50ec0d1d3f33aa8aedd28ee0c931a7e3283e15fa7ff881a3c06"} Oct 30 21:35:32.427803 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:32.427802 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" event={"ID":"a8c67e3a-dafa-4ed4-b5e3-f1b341427687","Type":"ContainerStarted","Data":"eccb01b1d39420aa8960aafbe0dda28508606acff74dc191d13627699ca5da4c"} Oct 30 21:35:32.448256 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:32.448204 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podStartSLOduration=1.125923998 podStartE2EDuration="1.448189377s" totalImagesPullingTime="322.265379ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:32.447804256 +0000 UTC m=+401.043620934" watchObservedRunningTime="2026-10-30 21:35:32.448189377 +0000 UTC m=+401.044006036" Oct 30 21:35:32.604456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:32.604425 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:35:34.304220 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:34.304180 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" Oct 30 21:35:34.316504 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:34.316478 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" Oct 30 21:35:34.448484 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:34.448456 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" Oct 30 21:35:35.254835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:35.254798 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:37.000689 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:37.000633 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" Oct 30 21:35:37.013126 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:37.013099 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" Oct 30 21:35:37.463911 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:37.463882 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" Oct 30 21:35:40.453384 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:40.453349 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" Oct 30 21:35:40.466034 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:40.466006 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" Oct 30 21:35:40.479480 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:40.479454 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" Oct 30 21:35:40.621059 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:40.621017 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:40.731108 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:40.731026 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:41.604564 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:41.604528 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:35:41.616571 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:41.616545 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:35:42.488723 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:42.488690 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:35:52.408049 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:52.408015 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:52.444951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:52.444928 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-lvkw5"] Oct 30 21:35:52.451566 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:52.451548 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-lvkw5" Oct 30 21:35:52.459298 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:52.457965 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-d4vm2\"" Oct 30 21:35:52.459298 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:52.458038 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 21:35:52.465009 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:52.463719 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-lvkw5"] Oct 30 21:35:52.528291 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:52.528257 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:52.553242 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:52.553211 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 21:35:52.553242 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:52.553239 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 21:35:52.553413 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:52.553270 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 21:35:52.634442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:52.634416 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93fc3e11-cb85-4381-85aa-c7b1abddbfb1-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-lvkw5\" (UID: \"93fc3e11-cb85-4381-85aa-c7b1abddbfb1\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-lvkw5" Oct 30 21:35:52.634575 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:52.634465 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c5ck\" (UniqueName: \"kubernetes.io/projected/93fc3e11-cb85-4381-85aa-c7b1abddbfb1-kube-api-access-2c5ck\") pod \"ai-gateway-controller-5f6dd4f69f-lvkw5\" (UID: \"93fc3e11-cb85-4381-85aa-c7b1abddbfb1\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-lvkw5" Oct 30 21:35:52.735072 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:52.734989 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93fc3e11-cb85-4381-85aa-c7b1abddbfb1-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-lvkw5\" (UID: \"93fc3e11-cb85-4381-85aa-c7b1abddbfb1\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-lvkw5" Oct 30 21:35:52.735072 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:52.735039 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c5ck\" (UniqueName: \"kubernetes.io/projected/93fc3e11-cb85-4381-85aa-c7b1abddbfb1-kube-api-access-2c5ck\") pod \"ai-gateway-controller-5f6dd4f69f-lvkw5\" (UID: \"93fc3e11-cb85-4381-85aa-c7b1abddbfb1\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-lvkw5" Oct 30 21:35:52.737518 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:52.737492 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93fc3e11-cb85-4381-85aa-c7b1abddbfb1-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-lvkw5\" (UID: \"93fc3e11-cb85-4381-85aa-c7b1abddbfb1\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-lvkw5" Oct 30 21:35:52.747379 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:52.747350 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c5ck\" (UniqueName: \"kubernetes.io/projected/93fc3e11-cb85-4381-85aa-c7b1abddbfb1-kube-api-access-2c5ck\") pod \"ai-gateway-controller-5f6dd4f69f-lvkw5\" (UID: \"93fc3e11-cb85-4381-85aa-c7b1abddbfb1\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-lvkw5" Oct 30 21:35:52.778369 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:52.778342 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-lvkw5" Oct 30 21:35:53.053137 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:53.053105 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-lvkw5"] Oct 30 21:35:53.055806 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:35:53.055781 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93fc3e11_cb85_4381_85aa_c7b1abddbfb1.slice/crio-327b75b919c61591871e99a35478b59fcb5a76ec17ea10d44f646b502d9904ac WatchSource:0}: Error finding container 327b75b919c61591871e99a35478b59fcb5a76ec17ea10d44f646b502d9904ac: Status 404 returned error can't find the container with id 327b75b919c61591871e99a35478b59fcb5a76ec17ea10d44f646b502d9904ac Oct 30 21:35:53.530643 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:53.530562 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-lvkw5" event={"ID":"93fc3e11-cb85-4381-85aa-c7b1abddbfb1","Type":"ContainerStarted","Data":"327b75b919c61591871e99a35478b59fcb5a76ec17ea10d44f646b502d9904ac"} Oct 30 21:35:56.881536 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:56.881500 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:57.269146 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.269027 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:57.290960 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.290930 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-hm54n"] Oct 30 21:35:57.294302 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.294287 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-hm54n" Oct 30 21:35:57.298489 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.298463 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 21:35:57.304389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.304364 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-hm54n"] Oct 30 21:35:57.307826 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.307801 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-b87ts"] Oct 30 21:35:57.311321 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.311302 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-b87ts" Oct 30 21:35:57.321744 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.321725 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-frsms\"" Oct 30 21:35:57.323369 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.323349 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-b87ts"] Oct 30 21:35:57.371140 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.371102 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:57.373659 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.373640 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006ecdc9-450d-4058-bd65-bebafca12ea0-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-hm54n\" (UID: \"006ecdc9-450d-4058-bd65-bebafca12ea0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-hm54n" Oct 30 21:35:57.373740 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.373681 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ff0bb8-b3e4-406a-a0b3-b20f27750e47-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-b87ts\" (UID: \"06ff0bb8-b3e4-406a-a0b3-b20f27750e47\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-b87ts" Oct 30 21:35:57.373850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.373831 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hxdv\" (UniqueName: \"kubernetes.io/projected/006ecdc9-450d-4058-bd65-bebafca12ea0-kube-api-access-7hxdv\") pod \"payload-pre-processing-7b98bb678c-hm54n\" (UID: \"006ecdc9-450d-4058-bd65-bebafca12ea0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-hm54n" Oct 30 21:35:57.373926 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.373914 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdknb\" (UniqueName: \"kubernetes.io/projected/06ff0bb8-b3e4-406a-a0b3-b20f27750e47-kube-api-access-tdknb\") pod \"payload-processing-55fbbbf9dc-b87ts\" (UID: \"06ff0bb8-b3e4-406a-a0b3-b20f27750e47\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-b87ts" Oct 30 21:35:57.474568 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.474522 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdknb\" (UniqueName: \"kubernetes.io/projected/06ff0bb8-b3e4-406a-a0b3-b20f27750e47-kube-api-access-tdknb\") pod \"payload-processing-55fbbbf9dc-b87ts\" (UID: \"06ff0bb8-b3e4-406a-a0b3-b20f27750e47\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-b87ts" Oct 30 21:35:57.474847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.474605 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006ecdc9-450d-4058-bd65-bebafca12ea0-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-hm54n\" (UID: \"006ecdc9-450d-4058-bd65-bebafca12ea0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-hm54n" Oct 30 21:35:57.474847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.474651 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ff0bb8-b3e4-406a-a0b3-b20f27750e47-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-b87ts\" (UID: \"06ff0bb8-b3e4-406a-a0b3-b20f27750e47\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-b87ts" Oct 30 21:35:57.474847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.474707 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hxdv\" (UniqueName: \"kubernetes.io/projected/006ecdc9-450d-4058-bd65-bebafca12ea0-kube-api-access-7hxdv\") pod \"payload-pre-processing-7b98bb678c-hm54n\" (UID: \"006ecdc9-450d-4058-bd65-bebafca12ea0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-hm54n" Oct 30 21:35:57.475853 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.475825 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ff0bb8-b3e4-406a-a0b3-b20f27750e47-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-b87ts\" (UID: \"06ff0bb8-b3e4-406a-a0b3-b20f27750e47\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-b87ts" Oct 30 21:35:57.476010 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.475991 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006ecdc9-450d-4058-bd65-bebafca12ea0-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-hm54n\" (UID: \"006ecdc9-450d-4058-bd65-bebafca12ea0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-hm54n" Oct 30 21:35:57.487499 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.487474 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hxdv\" (UniqueName: \"kubernetes.io/projected/006ecdc9-450d-4058-bd65-bebafca12ea0-kube-api-access-7hxdv\") pod \"payload-pre-processing-7b98bb678c-hm54n\" (UID: \"006ecdc9-450d-4058-bd65-bebafca12ea0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-hm54n" Oct 30 21:35:57.487778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.487739 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdknb\" (UniqueName: \"kubernetes.io/projected/06ff0bb8-b3e4-406a-a0b3-b20f27750e47-kube-api-access-tdknb\") pod \"payload-processing-55fbbbf9dc-b87ts\" (UID: \"06ff0bb8-b3e4-406a-a0b3-b20f27750e47\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-b87ts" Oct 30 21:35:57.550640 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.550562 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-lvkw5" event={"ID":"93fc3e11-cb85-4381-85aa-c7b1abddbfb1","Type":"ContainerStarted","Data":"8aaba66ec6f70d6ab2c47a01e5bb512f1fe815c7e11a14eeea12b811fb2d4607"} Oct 30 21:35:57.550809 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.550680 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-lvkw5" Oct 30 21:35:57.570393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.570348 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-lvkw5" podStartSLOduration=2.07697318 podStartE2EDuration="5.570335304s" totalImagesPullingTime="3.493362124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:57.568973686 +0000 UTC m=+426.164790360" watchObservedRunningTime="2026-10-30 21:35:57.570335304 +0000 UTC m=+426.166151964" Oct 30 21:35:57.608544 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.608526 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-hm54n" Oct 30 21:35:57.623654 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.623630 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-b87ts" Oct 30 21:35:57.891637 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.891608 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-hm54n"] Oct 30 21:35:57.894202 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:35:57.894169 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006ecdc9_450d_4058_bd65_bebafca12ea0.slice/crio-62496f61e8ed53cb7cdb675f57d4d2a3fc852f51bdb6b72c81c6da549435e259 WatchSource:0}: Error finding container 62496f61e8ed53cb7cdb675f57d4d2a3fc852f51bdb6b72c81c6da549435e259: Status 404 returned error can't find the container with id 62496f61e8ed53cb7cdb675f57d4d2a3fc852f51bdb6b72c81c6da549435e259 Oct 30 21:35:57.922805 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:57.922784 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-b87ts"] Oct 30 21:35:57.924690 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:35:57.924664 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ff0bb8_b3e4_406a_a0b3_b20f27750e47.slice/crio-f43ecb933453dce563cecbfff7962e432a870fc0612657bce52eb0ca25d8dc44 WatchSource:0}: Error finding container f43ecb933453dce563cecbfff7962e432a870fc0612657bce52eb0ca25d8dc44: Status 404 returned error can't find the container with id f43ecb933453dce563cecbfff7962e432a870fc0612657bce52eb0ca25d8dc44 Oct 30 21:35:58.558260 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:58.558213 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-hm54n" event={"ID":"006ecdc9-450d-4058-bd65-bebafca12ea0","Type":"ContainerStarted","Data":"62496f61e8ed53cb7cdb675f57d4d2a3fc852f51bdb6b72c81c6da549435e259"} Oct 30 21:35:58.560044 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:35:58.560014 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-b87ts" event={"ID":"06ff0bb8-b3e4-406a-a0b3-b20f27750e47","Type":"ContainerStarted","Data":"f43ecb933453dce563cecbfff7962e432a870fc0612657bce52eb0ca25d8dc44"} Oct 30 21:36:00.276625 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:36:00.276588 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:36:00.572211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:00.572171 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-hm54n" event={"ID":"006ecdc9-450d-4058-bd65-bebafca12ea0","Type":"ContainerStarted","Data":"2f6d29a73e20c245d14bcc1fe64916c2152f2380fb203336898efbc9ea026d42"} Oct 30 21:36:00.572401 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:00.572242 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-hm54n" Oct 30 21:36:00.573631 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:00.573606 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-b87ts" event={"ID":"06ff0bb8-b3e4-406a-a0b3-b20f27750e47","Type":"ContainerStarted","Data":"217ff5bc90a6e5153cd9024ecb49c9fa0a4553f45839a77119ef0633130e946d"} Oct 30 21:36:00.573730 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:00.573666 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-b87ts" Oct 30 21:36:00.593475 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:00.593434 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-hm54n" podStartSLOduration=1.332165923 podStartE2EDuration="3.593422282s" totalImagesPullingTime="2.261256359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:00.591742397 +0000 UTC m=+429.187559078" watchObservedRunningTime="2026-10-30 21:36:00.593422282 +0000 UTC m=+429.189238942" Oct 30 21:36:00.613100 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:00.613040 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-b87ts" podStartSLOduration=1.415213893 podStartE2EDuration="3.61302285s" totalImagesPullingTime="2.197808957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:00.610713043 +0000 UTC m=+429.206529728" watchObservedRunningTime="2026-10-30 21:36:00.61302285 +0000 UTC m=+429.208839510" Oct 30 21:36:07.025182 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.025141 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr"] Oct 30 21:36:07.028578 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.028558 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.055913 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.055881 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr"] Oct 30 21:36:07.060932 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.060906 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.061047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.060947 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.061047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.061040 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.061124 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.061066 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.061124 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.061116 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.061186 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.061136 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.061186 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.061161 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zxs5\" (UniqueName: \"kubernetes.io/projected/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-kube-api-access-7zxs5\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.061323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.061295 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.061360 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.061342 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.162200 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.162156 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.162200 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.162201 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.162448 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.162249 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.162448 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.162287 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.162448 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.162331 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.162448 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.162354 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.162448 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.162384 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.162448 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.162410 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.162844 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.162450 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zxs5\" (UniqueName: \"kubernetes.io/projected/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-kube-api-access-7zxs5\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.162844 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.162719 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.162844 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.162829 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.163010 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.162893 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.163010 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.162917 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.163292 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.163269 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.164601 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.164582 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.164689 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.164652 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.189862 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.189824 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.190873 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.190855 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zxs5\" (UniqueName: \"kubernetes.io/projected/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-kube-api-access-7zxs5\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-z2lsr\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.340806 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.340773 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:07.623181 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.623097 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr"] Oct 30 21:36:07.626889 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:36:07.626862 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493d5dd9_6bc8_43f2_b9ea_bf9989c20c2b.slice/crio-76e5a25cc766c804cc6bbcaa5caca50ee45e5faf2855f62e804e476fc69b42a5 WatchSource:0}: Error finding container 76e5a25cc766c804cc6bbcaa5caca50ee45e5faf2855f62e804e476fc69b42a5: Status 404 returned error can't find the container with id 76e5a25cc766c804cc6bbcaa5caca50ee45e5faf2855f62e804e476fc69b42a5 Oct 30 21:36:07.628883 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.628853 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:07.628956 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.628921 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:07.628956 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:07.628948 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:08.578429 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:08.578396 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-lvkw5" Oct 30 21:36:08.610822 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:08.610780 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" event={"ID":"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b","Type":"ContainerStarted","Data":"7556d3f99a0c67aa563c1911fa1b53ebbb0775b1569160dd71a41f9c8b83bcca"} Oct 30 21:36:08.610994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:08.610864 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" event={"ID":"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b","Type":"ContainerStarted","Data":"76e5a25cc766c804cc6bbcaa5caca50ee45e5faf2855f62e804e476fc69b42a5"} Oct 30 21:36:08.636696 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:08.636637 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" podStartSLOduration=2.636617423 podStartE2EDuration="2.636617423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:08.635434502 +0000 UTC m=+437.231251181" watchObservedRunningTime="2026-10-30 21:36:08.636617423 +0000 UTC m=+437.232434083" Oct 30 21:36:09.341349 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:09.341313 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:09.342840 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:09.342816 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:36:09.342968 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:09.342869 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" podUID="493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 21:36:10.342037 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:10.341998 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:36:10.342517 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:10.342061 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" podUID="493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 21:36:11.341985 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:11.341946 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:36:11.342162 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:11.342001 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" podUID="493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 21:36:11.580666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:11.580630 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-b87ts" Oct 30 21:36:11.580873 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:11.580706 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-hm54n" Oct 30 21:36:12.341714 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:12.341677 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:36:12.341916 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:12.341741 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" podUID="493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 21:36:13.341281 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:13.341245 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:36:13.341696 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:13.341300 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" podUID="493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 21:36:14.341606 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:14.341562 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:36:14.342203 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:14.341616 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" podUID="493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 21:36:15.792416 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:15.792378 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:15.792931 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:15.792694 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:15.793599 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:15.793577 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 21:36:21.929871 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:21.929831 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr"] Oct 30 21:36:21.934052 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:21.934030 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:21.950945 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:21.950915 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr"] Oct 30 21:36:22.009787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.009731 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66ktd\" (UniqueName: \"kubernetes.io/projected/fda6dedb-779c-49d8-8fcf-ba31f73402ae-kube-api-access-66ktd\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.009956 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.009794 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fda6dedb-779c-49d8-8fcf-ba31f73402ae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.009956 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.009898 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda6dedb-779c-49d8-8fcf-ba31f73402ae-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.009956 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.009952 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fda6dedb-779c-49d8-8fcf-ba31f73402ae-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.010052 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.010030 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fda6dedb-779c-49d8-8fcf-ba31f73402ae-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.010085 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.010059 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda6dedb-779c-49d8-8fcf-ba31f73402ae-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.010120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.010104 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fda6dedb-779c-49d8-8fcf-ba31f73402ae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.010185 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.010172 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fda6dedb-779c-49d8-8fcf-ba31f73402ae-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.010220 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.010196 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fda6dedb-779c-49d8-8fcf-ba31f73402ae-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.111014 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.110979 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fda6dedb-779c-49d8-8fcf-ba31f73402ae-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.111185 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.111029 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fda6dedb-779c-49d8-8fcf-ba31f73402ae-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.111185 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.111146 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda6dedb-779c-49d8-8fcf-ba31f73402ae-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.111255 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.111210 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fda6dedb-779c-49d8-8fcf-ba31f73402ae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.111255 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.111237 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fda6dedb-779c-49d8-8fcf-ba31f73402ae-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.111328 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.111264 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fda6dedb-779c-49d8-8fcf-ba31f73402ae-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.111328 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.111306 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66ktd\" (UniqueName: \"kubernetes.io/projected/fda6dedb-779c-49d8-8fcf-ba31f73402ae-kube-api-access-66ktd\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.111427 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.111338 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fda6dedb-779c-49d8-8fcf-ba31f73402ae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.111427 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.111399 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda6dedb-779c-49d8-8fcf-ba31f73402ae-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.111525 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.111435 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fda6dedb-779c-49d8-8fcf-ba31f73402ae-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.111525 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.111499 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda6dedb-779c-49d8-8fcf-ba31f73402ae-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.111732 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.111711 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda6dedb-779c-49d8-8fcf-ba31f73402ae-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.111897 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.111867 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fda6dedb-779c-49d8-8fcf-ba31f73402ae-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.112201 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.112176 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fda6dedb-779c-49d8-8fcf-ba31f73402ae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.113467 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.113447 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fda6dedb-779c-49d8-8fcf-ba31f73402ae-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.113795 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.113775 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fda6dedb-779c-49d8-8fcf-ba31f73402ae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.124491 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.124463 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fda6dedb-779c-49d8-8fcf-ba31f73402ae-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.124573 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.124528 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66ktd\" (UniqueName: \"kubernetes.io/projected/fda6dedb-779c-49d8-8fcf-ba31f73402ae-kube-api-access-66ktd\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7wxsr\" (UID: \"fda6dedb-779c-49d8-8fcf-ba31f73402ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.248389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.248312 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:22.534807 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.534776 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr"] Oct 30 21:36:22.537401 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:36:22.537371 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda6dedb_779c_49d8_8fcf_ba31f73402ae.slice/crio-a44f070876cacb519aa0702395e0005ab7ee7c674d0322f98b48a7b198e36e26 WatchSource:0}: Error finding container a44f070876cacb519aa0702395e0005ab7ee7c674d0322f98b48a7b198e36e26: Status 404 returned error can't find the container with id a44f070876cacb519aa0702395e0005ab7ee7c674d0322f98b48a7b198e36e26 Oct 30 21:36:22.539864 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.539834 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:22.539926 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.539905 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:22.539959 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.539932 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:22.681967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.681899 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" event={"ID":"fda6dedb-779c-49d8-8fcf-ba31f73402ae","Type":"ContainerStarted","Data":"c097167031d82bf0ed2450370f520a3f61066d3a87c9d4d4c5050305dc4a7c2b"} Oct 30 21:36:22.681967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.681954 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" event={"ID":"fda6dedb-779c-49d8-8fcf-ba31f73402ae","Type":"ContainerStarted","Data":"a44f070876cacb519aa0702395e0005ab7ee7c674d0322f98b48a7b198e36e26"} Oct 30 21:36:22.709597 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:22.709545 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" podStartSLOduration=1.709530285 podStartE2EDuration="1.709530285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:22.708676559 +0000 UTC m=+451.304493244" watchObservedRunningTime="2026-10-30 21:36:22.709530285 +0000 UTC m=+451.305346944" Oct 30 21:36:23.248818 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:23.248786 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:23.250290 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:23.250265 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:36:23.250439 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:23.250312 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" podUID="fda6dedb-779c-49d8-8fcf-ba31f73402ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:36:24.008597 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:24.008559 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-lvkw5"] Oct 30 21:36:24.008815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:24.008793 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-lvkw5" podUID="93fc3e11-cb85-4381-85aa-c7b1abddbfb1" containerName="manager" containerID="cri-o://8aaba66ec6f70d6ab2c47a01e5bb512f1fe815c7e11a14eeea12b811fb2d4607" gracePeriod=30 Oct 30 21:36:24.249066 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:24.249032 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:36:24.249423 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:24.249090 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" podUID="fda6dedb-779c-49d8-8fcf-ba31f73402ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:36:24.302507 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:24.302483 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-lvkw5" Oct 30 21:36:24.435202 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:24.435172 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93fc3e11-cb85-4381-85aa-c7b1abddbfb1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93fc3e11-cb85-4381-85aa-c7b1abddbfb1-metrics-certs\") pod \"93fc3e11-cb85-4381-85aa-c7b1abddbfb1\" (UID: \"93fc3e11-cb85-4381-85aa-c7b1abddbfb1\") " Oct 30 21:36:24.435377 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:24.435231 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93fc3e11-cb85-4381-85aa-c7b1abddbfb1-kube-api-access-2c5ck\" (UniqueName: \"kubernetes.io/projected/93fc3e11-cb85-4381-85aa-c7b1abddbfb1-kube-api-access-2c5ck\") pod \"93fc3e11-cb85-4381-85aa-c7b1abddbfb1\" (UID: \"93fc3e11-cb85-4381-85aa-c7b1abddbfb1\") " Oct 30 21:36:24.437397 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:24.437366 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93fc3e11-cb85-4381-85aa-c7b1abddbfb1-metrics-certs" pod "93fc3e11-cb85-4381-85aa-c7b1abddbfb1" (UID: "93fc3e11-cb85-4381-85aa-c7b1abddbfb1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:36:24.437511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:24.437436 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93fc3e11-cb85-4381-85aa-c7b1abddbfb1-kube-api-access-2c5ck" pod "93fc3e11-cb85-4381-85aa-c7b1abddbfb1" (UID: "93fc3e11-cb85-4381-85aa-c7b1abddbfb1"). InnerVolumeSpecName "kube-api-access-2c5ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:24.536771 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:24.536674 5801 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93fc3e11-cb85-4381-85aa-c7b1abddbfb1-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:36:24.536771 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:24.536703 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c5ck\" (UniqueName: \"kubernetes.io/projected/93fc3e11-cb85-4381-85aa-c7b1abddbfb1-kube-api-access-2c5ck\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:36:24.693193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:24.693161 5801 generic.go:356] "Generic (PLEG): container finished" podID="93fc3e11-cb85-4381-85aa-c7b1abddbfb1" containerID="8aaba66ec6f70d6ab2c47a01e5bb512f1fe815c7e11a14eeea12b811fb2d4607" exitCode=0 Oct 30 21:36:24.693366 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:24.693207 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-lvkw5" Oct 30 21:36:24.693366 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:24.693251 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-lvkw5" event={"ID":"93fc3e11-cb85-4381-85aa-c7b1abddbfb1","Type":"ContainerDied","Data":"8aaba66ec6f70d6ab2c47a01e5bb512f1fe815c7e11a14eeea12b811fb2d4607"} Oct 30 21:36:24.693366 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:24.693304 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-lvkw5" event={"ID":"93fc3e11-cb85-4381-85aa-c7b1abddbfb1","Type":"ContainerDied","Data":"327b75b919c61591871e99a35478b59fcb5a76ec17ea10d44f646b502d9904ac"} Oct 30 21:36:24.693366 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:24.693324 5801 scope.go:122] "RemoveContainer" containerID="8aaba66ec6f70d6ab2c47a01e5bb512f1fe815c7e11a14eeea12b811fb2d4607" Oct 30 21:36:24.703446 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:24.703431 5801 scope.go:122] "RemoveContainer" containerID="8aaba66ec6f70d6ab2c47a01e5bb512f1fe815c7e11a14eeea12b811fb2d4607" Oct 30 21:36:24.703707 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:36:24.703686 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aaba66ec6f70d6ab2c47a01e5bb512f1fe815c7e11a14eeea12b811fb2d4607\": container with ID starting with 8aaba66ec6f70d6ab2c47a01e5bb512f1fe815c7e11a14eeea12b811fb2d4607 not found: ID does not exist" containerID="8aaba66ec6f70d6ab2c47a01e5bb512f1fe815c7e11a14eeea12b811fb2d4607" Oct 30 21:36:24.703808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:24.703721 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aaba66ec6f70d6ab2c47a01e5bb512f1fe815c7e11a14eeea12b811fb2d4607"} err="failed to get container status \"8aaba66ec6f70d6ab2c47a01e5bb512f1fe815c7e11a14eeea12b811fb2d4607\": rpc error: code = NotFound desc = could not find container \"8aaba66ec6f70d6ab2c47a01e5bb512f1fe815c7e11a14eeea12b811fb2d4607\": container with ID starting with 8aaba66ec6f70d6ab2c47a01e5bb512f1fe815c7e11a14eeea12b811fb2d4607 not found: ID does not exist" Oct 30 21:36:24.716156 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:24.716134 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-lvkw5"] Oct 30 21:36:24.720167 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:24.720149 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-lvkw5"] Oct 30 21:36:25.249270 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:25.249236 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:36:25.249680 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:25.249291 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" podUID="fda6dedb-779c-49d8-8fcf-ba31f73402ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:36:25.522045 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:25.521971 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93fc3e11-cb85-4381-85aa-c7b1abddbfb1" path="/var/lib/kubelet/pods/93fc3e11-cb85-4381-85aa-c7b1abddbfb1/volumes" Oct 30 21:36:25.878487 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:25.878450 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:26.248976 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:26.248887 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:36:26.248976 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:26.248940 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" podUID="fda6dedb-779c-49d8-8fcf-ba31f73402ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:36:26.271777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:26.267106 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:27.249542 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:27.249507 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:36:27.249706 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:27.249565 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" podUID="fda6dedb-779c-49d8-8fcf-ba31f73402ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:36:27.974787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:27.974240 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:28.083634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:28.083587 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:28.248944 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:28.248865 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:36:28.248944 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:28.248922 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" podUID="fda6dedb-779c-49d8-8fcf-ba31f73402ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:36:29.248730 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:29.248693 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:36:29.249159 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:29.248777 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" podUID="fda6dedb-779c-49d8-8fcf-ba31f73402ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:36:29.583465 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:29.583426 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:29.694613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:29.694574 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:30.278524 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:36:30.278480 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:36:30.703384 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:30.703351 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:30.704096 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:30.704048 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:30.704948 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:30.704926 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" Oct 30 21:36:36.957616 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:36.957529 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm"] Oct 30 21:36:36.958051 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:36.957950 5801 cpu_manager.go:414] "containerMap: removing container" podUID="93fc3e11-cb85-4381-85aa-c7b1abddbfb1" containerName="manager" Oct 30 21:36:36.958051 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:36.957963 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93fc3e11-cb85-4381-85aa-c7b1abddbfb1" containerName="manager" Oct 30 21:36:36.958051 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:36.958020 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="93fc3e11-cb85-4381-85aa-c7b1abddbfb1" containerName="manager" Oct 30 21:36:36.961356 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:36.961332 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:36.980360 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:36.980328 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm"] Oct 30 21:36:37.074402 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.074362 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.074578 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.074412 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.074578 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.074492 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.074578 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.074523 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.074685 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.074611 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.074685 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.074660 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgjl5\" (UniqueName: \"kubernetes.io/projected/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-kube-api-access-xgjl5\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.074685 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.074679 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.074840 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.074736 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.074840 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.074826 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.175785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.175719 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.175785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.175789 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.176033 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.175814 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.176033 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.175856 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.176033 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.175911 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgjl5\" (UniqueName: \"kubernetes.io/projected/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-kube-api-access-xgjl5\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.176033 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.175936 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.176033 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.175980 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.176278 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.176048 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.176278 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.176107 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.176278 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.176204 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.176278 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.176241 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.176486 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.176313 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.176548 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.176517 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.176628 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.176604 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.178326 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.178306 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.178578 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.178549 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.191121 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.191090 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.191310 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.191289 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgjl5\" (UniqueName: \"kubernetes.io/projected/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-kube-api-access-xgjl5\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jmcsm\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.277378 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.277280 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:37.557807 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.557607 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm"] Oct 30 21:36:37.560311 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:36:37.560287 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f8dce17_6c39_43d9_9a59_0f2cac4cb7e9.slice/crio-b7cdc347562175f31758816c333e3ea3b0c23ac5a4acd661b54ca147525c64d1 WatchSource:0}: Error finding container b7cdc347562175f31758816c333e3ea3b0c23ac5a4acd661b54ca147525c64d1: Status 404 returned error can't find the container with id b7cdc347562175f31758816c333e3ea3b0c23ac5a4acd661b54ca147525c64d1 Oct 30 21:36:37.562431 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.562396 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:37.562516 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.562477 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:37.562516 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.562510 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:37.766681 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.766642 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" event={"ID":"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9","Type":"ContainerStarted","Data":"491aa761acede556be776655a81f800a3967ef7b917a11436202c5d2243e9e99"} Oct 30 21:36:37.766889 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.766712 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" event={"ID":"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9","Type":"ContainerStarted","Data":"b7cdc347562175f31758816c333e3ea3b0c23ac5a4acd661b54ca147525c64d1"} Oct 30 21:36:37.794657 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:37.794610 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" podStartSLOduration=1.7945956170000001 podStartE2EDuration="1.794595617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:37.793936127 +0000 UTC m=+466.389752808" watchObservedRunningTime="2026-10-30 21:36:37.794595617 +0000 UTC m=+466.390412294" Oct 30 21:36:38.277987 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:38.277944 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:38.279566 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:38.279533 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:36:38.279695 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:38.279587 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" podUID="9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 21:36:39.278162 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:39.278127 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:36:39.278640 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:39.278185 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" podUID="9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 21:36:40.278227 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:40.278182 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:36:40.278645 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:40.278268 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" podUID="9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 21:36:41.278298 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:41.278257 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:36:41.278822 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:41.278353 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" podUID="9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 21:36:42.278146 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:42.278101 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:36:42.278336 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:42.278165 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" podUID="9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 21:36:43.278116 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:43.278082 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:36:43.278275 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:43.278168 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" podUID="9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 21:36:44.277993 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:44.277947 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:36:44.278403 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:44.278025 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" podUID="9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 21:36:45.691030 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:45.691001 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:45.691496 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:45.691423 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:45.692179 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:45.692161 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 21:36:52.391158 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:52.391120 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:52.501133 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:52.501088 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:52.817780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:52.817736 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl"] Oct 30 21:36:52.823264 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:52.823244 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:52.832650 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:52.832630 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0fa864-openshift-default-dockercfg-bwzs2\"" Oct 30 21:36:52.850819 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:52.850792 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl"] Oct 30 21:36:52.909992 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:52.909954 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-workload-certs\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:52.910171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:52.910019 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f258bc9-d2fd-446a-9154-36caf951e081-istio-podinfo\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:52.910171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:52.910099 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-workload-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:52.910171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:52.910148 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-istio-envoy\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:52.910339 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:52.910267 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f258bc9-d2fd-446a-9154-36caf951e081-istiod-ca-cert\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:52.910339 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:52.910292 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9c4s\" (UniqueName: \"kubernetes.io/projected/0f258bc9-d2fd-446a-9154-36caf951e081-kube-api-access-c9c4s\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:52.910339 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:52.910315 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-credential-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:52.910339 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:52.910333 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f258bc9-d2fd-446a-9154-36caf951e081-istio-token\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:52.910545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:52.910477 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-istio-data\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:52.916137 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:52.916113 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:52.982246 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:52.982216 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n"] Oct 30 21:36:52.986901 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:52.986881 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.001658 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.001640 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-caa3dc-openshift-default-dockercfg-5kw62\"" Oct 30 21:36:53.009054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.009031 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n"] Oct 30 21:36:53.011040 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.011017 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-istio-data\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:53.011138 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.011067 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-workload-certs\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:53.011138 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.011132 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-data\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.011287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.011176 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-token\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.011287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.011204 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-credential-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.011287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.011229 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-envoy\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.011287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.011260 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f258bc9-d2fd-446a-9154-36caf951e081-istio-podinfo\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:53.011287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.011286 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-workload-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:53.014791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.011632 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-workload-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:53.014791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.011723 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-workload-certs\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:53.014791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.011798 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-podinfo\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.014791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.011841 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-istio-envoy\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:53.014791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.011901 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f258bc9-d2fd-446a-9154-36caf951e081-istiod-ca-cert\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:53.014791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.011926 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9c4s\" (UniqueName: \"kubernetes.io/projected/0f258bc9-d2fd-446a-9154-36caf951e081-kube-api-access-c9c4s\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:53.014791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.011955 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-workload-certs\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.014791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.011986 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-istio-data\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:53.014791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.011988 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istiod-ca-cert\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.014791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.012032 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-credential-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:53.014791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.012070 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-workload-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.014791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.012097 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f258bc9-d2fd-446a-9154-36caf951e081-istio-token\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:53.014791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.012127 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbh28\" (UniqueName: \"kubernetes.io/projected/f205247c-2c3c-4dc3-a3d2-b46066918ae3-kube-api-access-nbh28\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.014791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.012494 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-credential-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:53.014791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.012521 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f258bc9-d2fd-446a-9154-36caf951e081-istiod-ca-cert\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:53.014791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.013871 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f258bc9-d2fd-446a-9154-36caf951e081-istio-podinfo\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:53.014791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.013880 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-istio-envoy\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:53.031215 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.031195 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9c4s\" (UniqueName: \"kubernetes.io/projected/0f258bc9-d2fd-446a-9154-36caf951e081-kube-api-access-c9c4s\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:53.046294 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.046265 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f258bc9-d2fd-446a-9154-36caf951e081-istio-token\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-275jl\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:53.071677 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.071588 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:53.109808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.109781 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps"] Oct 30 21:36:53.113478 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.113457 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-workload-certs\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.113587 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.113491 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istiod-ca-cert\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.113587 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.113519 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-workload-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.113587 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.113553 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbh28\" (UniqueName: \"kubernetes.io/projected/f205247c-2c3c-4dc3-a3d2-b46066918ae3-kube-api-access-nbh28\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.113700 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.113647 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-data\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.113700 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.113685 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-token\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.113854 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.113714 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-credential-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.113854 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.113740 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-envoy\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.113854 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.113797 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-podinfo\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.114013 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.113889 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-workload-certs\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.114013 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.113948 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-workload-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.114162 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.114142 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-data\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.114222 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.114175 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-credential-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.114282 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.114256 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istiod-ca-cert\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.114815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.114795 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.116202 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.116180 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-envoy\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.116440 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.116417 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-podinfo\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.129810 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.129786 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-7dfff0-openshift-default-dockercfg-lrx4l\"" Oct 30 21:36:53.136999 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.136977 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-token\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.137143 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.137128 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:53.138048 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.138030 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbh28\" (UniqueName: \"kubernetes.io/projected/f205247c-2c3c-4dc3-a3d2-b46066918ae3-kube-api-access-nbh28\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.138150 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.138046 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps"] Oct 30 21:36:53.213632 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.213582 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:53.214938 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.214908 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56r4q\" (UniqueName: \"kubernetes.io/projected/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-kube-api-access-56r4q\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.215068 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.214953 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.215068 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.215050 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.215196 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.215157 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.215259 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.215196 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.215259 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.215245 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.215379 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.215282 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.215379 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.215315 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.215379 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.215364 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.302776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.301935 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:53.320257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.316339 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.320257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.316399 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.320257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.316441 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56r4q\" (UniqueName: \"kubernetes.io/projected/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-kube-api-access-56r4q\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.320257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.316465 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.320257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.316496 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.320257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.316569 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.320257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.316599 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.320257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.316640 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.320257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.318945 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.320257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.319013 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.320257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.319075 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.320257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.319504 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.322695 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.322464 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.328974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.323074 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.328974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.328846 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:53.332899 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.332830 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.352514 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.352294 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.356170 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.356108 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.359386 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.359360 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz"] Oct 30 21:36:53.359939 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.359913 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56r4q\" (UniqueName: \"kubernetes.io/projected/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-kube-api-access-56r4q\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.364547 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.364526 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz"] Oct 30 21:36:53.364664 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.364638 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.374053 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.374033 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-1a81ae-openshift-default-dockercfg-mbfh7\"" Oct 30 21:36:53.424366 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.424338 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-envoy\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.424695 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.424377 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-credential-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.424695 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.424484 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-data\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.424695 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.424512 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42crw\" (UniqueName: \"kubernetes.io/projected/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-kube-api-access-42crw\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.424695 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.424577 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-workload-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.424695 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.424627 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istiod-ca-cert\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.424695 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.424645 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-workload-certs\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.424695 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.424660 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-podinfo\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.424981 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.424772 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-token\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.436219 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.436198 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:53.460154 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.460125 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl"] Oct 30 21:36:53.461790 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:36:53.461733 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f258bc9_d2fd_446a_9154_36caf951e081.slice/crio-0cef7a4ed47262749f6760a23375cfae663aeeae2988aae6c4a8c66fcb9e5407 WatchSource:0}: Error finding container 0cef7a4ed47262749f6760a23375cfae663aeeae2988aae6c4a8c66fcb9e5407: Status 404 returned error can't find the container with id 0cef7a4ed47262749f6760a23375cfae663aeeae2988aae6c4a8c66fcb9e5407 Oct 30 21:36:53.464218 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.464189 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:53.464315 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.464296 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:53.464384 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.464367 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:53.488812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.488090 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:53.525903 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.525878 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-envoy\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.526016 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.525915 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-credential-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.526016 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.525953 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-data\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.526016 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.525971 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42crw\" (UniqueName: \"kubernetes.io/projected/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-kube-api-access-42crw\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.526016 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.525999 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-workload-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.526206 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.526048 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istiod-ca-cert\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.526206 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.526064 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-workload-certs\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.526206 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.526085 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-podinfo\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.526206 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.526126 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-token\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.526404 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.526347 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-credential-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.526476 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.526449 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-data\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.526652 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.526631 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-workload-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.526846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.526822 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istiod-ca-cert\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.527034 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.527018 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-workload-certs\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.528281 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.528260 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-envoy\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.528670 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.528652 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-podinfo\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.567798 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.567748 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42crw\" (UniqueName: \"kubernetes.io/projected/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-kube-api-access-42crw\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.568259 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.568235 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-token\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.643534 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.643501 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n"] Oct 30 21:36:53.643795 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:36:53.643767 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf205247c_2c3c_4dc3_a3d2_b46066918ae3.slice/crio-6f74504589a577027e94db6ecb17a046e0fc77cace1429df4b932b40b7b3326a WatchSource:0}: Error finding container 6f74504589a577027e94db6ecb17a046e0fc77cace1429df4b932b40b7b3326a: Status 404 returned error can't find the container with id 6f74504589a577027e94db6ecb17a046e0fc77cace1429df4b932b40b7b3326a Oct 30 21:36:53.648554 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.646684 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:53.648554 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.647620 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:53.648554 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.647678 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:53.653094 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.651324 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm"] Oct 30 21:36:53.660325 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.655593 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:53.660429 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.660402 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.670256 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.670213 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-000f64-openshift-default-dockercfg-8l2xx\"" Oct 30 21:36:53.672852 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.672064 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm"] Oct 30 21:36:53.680765 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.680731 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:53.728743 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.728711 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-credential-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.728894 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.728802 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ea6c627-8885-490d-88ee-10befd58f673-istio-podinfo\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.728894 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.728865 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tl2b\" (UniqueName: \"kubernetes.io/projected/9ea6c627-8885-490d-88ee-10befd58f673-kube-api-access-8tl2b\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.729027 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.728897 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ea6c627-8885-490d-88ee-10befd58f673-istiod-ca-cert\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.729027 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.728944 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-workload-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.729027 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.729001 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-workload-certs\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.729184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.729028 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-istio-data\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.729184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.729057 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ea6c627-8885-490d-88ee-10befd58f673-istio-token\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.729184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.729148 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-istio-envoy\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.739478 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.733229 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps"] Oct 30 21:36:53.745623 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.742307 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:53.745623 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.742389 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:53.745623 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.742432 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:53.776127 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.776092 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:53.865847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.834934 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-istio-envoy\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.865847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.835014 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-credential-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.865847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.835048 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ea6c627-8885-490d-88ee-10befd58f673-istio-podinfo\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.865847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.835100 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tl2b\" (UniqueName: \"kubernetes.io/projected/9ea6c627-8885-490d-88ee-10befd58f673-kube-api-access-8tl2b\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.865847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.835134 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ea6c627-8885-490d-88ee-10befd58f673-istiod-ca-cert\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.865847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.835172 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-workload-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.865847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.835225 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-workload-certs\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.865847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.835249 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-istio-data\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.865847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.835276 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ea6c627-8885-490d-88ee-10befd58f673-istio-token\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.865847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.864803 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ea6c627-8885-490d-88ee-10befd58f673-istiod-ca-cert\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.865847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.865102 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-workload-certs\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.865847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.865259 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-workload-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.865847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.865515 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-credential-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.866679 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.865897 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-istio-data\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.869142 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.869111 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ea6c627-8885-490d-88ee-10befd58f673-istio-podinfo\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.870990 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.870954 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ea6c627-8885-490d-88ee-10befd58f673-istio-token\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.872598 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.872572 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-istio-envoy\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.882813 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.882236 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tl2b\" (UniqueName: \"kubernetes.io/projected/9ea6c627-8885-490d-88ee-10befd58f673-kube-api-access-8tl2b\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.885125 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.884974 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" event={"ID":"8aa256c9-7e49-4144-9c43-91c8cdca8a7a","Type":"ContainerStarted","Data":"20b3f53a9fbe0f0fd7acc1ea47040e4f48bcce2df8905d05e16b7b9d35da496e"} Oct 30 21:36:53.886660 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.886637 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" event={"ID":"0f258bc9-d2fd-446a-9154-36caf951e081","Type":"ContainerStarted","Data":"76fe2b8ce6f3ea3a2d1c7ecec5e9b7bd17b1b9a550a56410f745684e91c3ce96"} Oct 30 21:36:53.886774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.886681 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" event={"ID":"0f258bc9-d2fd-446a-9154-36caf951e081","Type":"ContainerStarted","Data":"0cef7a4ed47262749f6760a23375cfae663aeeae2988aae6c4a8c66fcb9e5407"} Oct 30 21:36:53.888514 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.888494 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" event={"ID":"f205247c-2c3c-4dc3-a3d2-b46066918ae3","Type":"ContainerStarted","Data":"3e4a2b5342a6f8be4d987bb6df54b325b28bfe3f06f18725669bc5941da34a73"} Oct 30 21:36:53.888617 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.888529 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" event={"ID":"f205247c-2c3c-4dc3-a3d2-b46066918ae3","Type":"ContainerStarted","Data":"6f74504589a577027e94db6ecb17a046e0fc77cace1429df4b932b40b7b3326a"} Oct 30 21:36:53.912486 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.912200 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" podStartSLOduration=1.912183861 podStartE2EDuration="1.912183861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:53.909901835 +0000 UTC m=+482.505718506" watchObservedRunningTime="2026-10-30 21:36:53.912183861 +0000 UTC m=+482.508000522" Oct 30 21:36:53.935803 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.935728 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" podStartSLOduration=1.9357089790000002 podStartE2EDuration="1.935708979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:53.934597291 +0000 UTC m=+482.530413948" watchObservedRunningTime="2026-10-30 21:36:53.935708979 +0000 UTC m=+482.531525640" Oct 30 21:36:53.974532 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.974505 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz"] Oct 30 21:36:53.977422 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.977357 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:53.977588 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:36:53.977558 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec34f7ef_9765_4b17_bdfe_cefd1f2a37b2.slice/crio-9dd412d8651bfb13ad9d03222019cb9951d8b49e138dc2e9cc4597376621dd2e WatchSource:0}: Error finding container 9dd412d8651bfb13ad9d03222019cb9951d8b49e138dc2e9cc4597376621dd2e: Status 404 returned error can't find the container with id 9dd412d8651bfb13ad9d03222019cb9951d8b49e138dc2e9cc4597376621dd2e Oct 30 21:36:53.980566 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.980529 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:53.980648 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.980612 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:53.980783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:53.980659 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:54.138292 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:54.138260 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:54.143361 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:54.143338 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:54.277715 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:54.277684 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm"] Oct 30 21:36:54.279159 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:36:54.279113 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea6c627_8885_490d_88ee_10befd58f673.slice/crio-4092d61b2c12339c43ac17e13e617f930f65252c2c403298e3de61064cd77d43 WatchSource:0}: Error finding container 4092d61b2c12339c43ac17e13e617f930f65252c2c403298e3de61064cd77d43: Status 404 returned error can't find the container with id 4092d61b2c12339c43ac17e13e617f930f65252c2c403298e3de61064cd77d43 Oct 30 21:36:54.281815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:54.281783 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:54.281882 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:54.281846 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:54.281882 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:54.281874 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:36:54.302904 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:54.302875 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:54.308145 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:54.308126 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:54.895162 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:54.895127 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" event={"ID":"8aa256c9-7e49-4144-9c43-91c8cdca8a7a","Type":"ContainerStarted","Data":"5eff4855aa7a6356c524849160e4af14d893fcf75713e393ff74f13aaffd3211"} Oct 30 21:36:54.896717 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:54.896692 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" event={"ID":"9ea6c627-8885-490d-88ee-10befd58f673","Type":"ContainerStarted","Data":"26f388b059480b1f1ecc6f86cea4c809d8d4230370c30f25da41e897eca4993a"} Oct 30 21:36:54.896842 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:54.896739 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" event={"ID":"9ea6c627-8885-490d-88ee-10befd58f673","Type":"ContainerStarted","Data":"4092d61b2c12339c43ac17e13e617f930f65252c2c403298e3de61064cd77d43"} Oct 30 21:36:54.898196 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:54.898172 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" event={"ID":"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2","Type":"ContainerStarted","Data":"30c565eb326a7c2d89560e7c9e91fd808763b066617b64f4335aa365f6f103b5"} Oct 30 21:36:54.898299 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:54.898216 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" event={"ID":"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2","Type":"ContainerStarted","Data":"9dd412d8651bfb13ad9d03222019cb9951d8b49e138dc2e9cc4597376621dd2e"} Oct 30 21:36:54.898874 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:54.898855 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:54.898979 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:54.898892 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:54.899368 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:54.899348 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:36:54.899443 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:54.899400 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:36:54.919792 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:54.919739 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" podStartSLOduration=1.9197287360000002 podStartE2EDuration="1.919728736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:54.917791817 +0000 UTC m=+483.513608499" watchObservedRunningTime="2026-10-30 21:36:54.919728736 +0000 UTC m=+483.515545396" Oct 30 21:36:54.940908 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:54.940863 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" podStartSLOduration=1.940849625 podStartE2EDuration="1.940849625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:54.940302779 +0000 UTC m=+483.536119466" watchObservedRunningTime="2026-10-30 21:36:54.940849625 +0000 UTC m=+483.536666286" Oct 30 21:36:54.977725 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:54.977699 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:54.986868 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:54.986803 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" podStartSLOduration=1.986783098 podStartE2EDuration="1.986783098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:54.984909838 +0000 UTC m=+483.580726521" watchObservedRunningTime="2026-10-30 21:36:54.986783098 +0000 UTC m=+483.582599802" Oct 30 21:36:55.436366 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:55.436337 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:55.441787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:55.441743 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:55.681694 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:55.681654 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:55.686877 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:55.686817 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:55.904782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:55.904726 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:55.904782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:55.904789 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:55.905440 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:55.905420 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:36:55.905531 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:55.905467 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:36:55.983380 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:55.983303 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:56.246045 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:56.245959 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:56.364556 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:56.364520 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:56.519641 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:56.519552 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:56.637861 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:56.637808 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:56.756519 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:56.756487 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:56.909404 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:56.909378 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:56.910386 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:56.910367 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:36:56.961436 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:56.961405 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:57.529417 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:57.529386 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:57.668133 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:57.668103 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:57.858057 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:57.858024 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:57.987367 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:57.987337 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:58.101076 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:58.101048 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:58.362319 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:58.362279 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:58.478133 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:58.478099 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:58.811236 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:58.811202 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:58.966115 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:58.966073 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:59.109949 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:59.109849 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:59.225168 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:59.225139 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:59.588289 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:59.588259 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:59.713160 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:36:59.713126 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:00.277637 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:37:00.277600 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:37:00.542154 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:00.542048 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:00.745567 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:00.745535 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:00.881235 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:00.881201 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:01.247123 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:01.247046 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:01.360031 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:01.359999 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79"] Oct 30 21:37:01.363981 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:01.363960 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79" Oct 30 21:37:01.386931 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:01.386888 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79"] Oct 30 21:37:01.393247 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:01.393215 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:01.426277 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:01.426257 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-7dfff0-dockercfg-fxbnc\"" Oct 30 21:37:01.433790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:01.433736 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b2ea58-b940-4434-8b89-c3d7c0bc3cba-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79\" (UID: \"c7b2ea58-b940-4434-8b89-c3d7c0bc3cba\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79" Oct 30 21:37:01.434090 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:01.434062 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhjp4\" (UniqueName: \"kubernetes.io/projected/c7b2ea58-b940-4434-8b89-c3d7c0bc3cba-kube-api-access-qhjp4\") pod \"payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79\" (UID: \"c7b2ea58-b940-4434-8b89-c3d7c0bc3cba\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79" Oct 30 21:37:01.460776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:01.456841 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-7dfff0\"" Oct 30 21:37:01.546108 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:01.544417 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b2ea58-b940-4434-8b89-c3d7c0bc3cba-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79\" (UID: \"c7b2ea58-b940-4434-8b89-c3d7c0bc3cba\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79" Oct 30 21:37:01.546108 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:01.544540 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhjp4\" (UniqueName: \"kubernetes.io/projected/c7b2ea58-b940-4434-8b89-c3d7c0bc3cba-kube-api-access-qhjp4\") pod \"payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79\" (UID: \"c7b2ea58-b940-4434-8b89-c3d7c0bc3cba\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79" Oct 30 21:37:01.546108 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:01.545822 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b2ea58-b940-4434-8b89-c3d7c0bc3cba-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79\" (UID: \"c7b2ea58-b940-4434-8b89-c3d7c0bc3cba\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79" Oct 30 21:37:01.560889 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:01.560861 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhjp4\" (UniqueName: \"kubernetes.io/projected/c7b2ea58-b940-4434-8b89-c3d7c0bc3cba-kube-api-access-qhjp4\") pod \"payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79\" (UID: \"c7b2ea58-b940-4434-8b89-c3d7c0bc3cba\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79" Oct 30 21:37:01.677528 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:01.677501 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79" Oct 30 21:37:01.784000 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:01.783965 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:01.907290 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:01.907258 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw"] Oct 30 21:37:01.911713 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:01.911696 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw" Oct 30 21:37:01.923826 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:01.923798 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw"] Oct 30 21:37:01.923997 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:01.923976 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-1a81ae\"" Oct 30 21:37:01.932386 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:01.932229 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:01.960207 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:01.960175 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79"] Oct 30 21:37:01.964969 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:37:01.964939 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b2ea58_b940_4434_8b89_c3d7c0bc3cba.slice/crio-bf97457de0ac059e9debfd7ca7550726c96c066b3c547fdb708830bd400f2366 WatchSource:0}: Error finding container bf97457de0ac059e9debfd7ca7550726c96c066b3c547fdb708830bd400f2366: Status 404 returned error can't find the container with id bf97457de0ac059e9debfd7ca7550726c96c066b3c547fdb708830bd400f2366 Oct 30 21:37:01.972205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:01.972178 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-1a81ae-dockercfg-tcrfr\"" Oct 30 21:37:02.050202 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:02.050041 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56xsr\" (UniqueName: \"kubernetes.io/projected/abc6bc4f-b6fe-49aa-9f79-41da164ffa08-kube-api-access-56xsr\") pod \"payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw\" (UID: \"abc6bc4f-b6fe-49aa-9f79-41da164ffa08\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw" Oct 30 21:37:02.050376 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:02.050350 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc6bc4f-b6fe-49aa-9f79-41da164ffa08-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw\" (UID: \"abc6bc4f-b6fe-49aa-9f79-41da164ffa08\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw" Oct 30 21:37:02.085101 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:02.085064 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:02.165876 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:02.160296 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc6bc4f-b6fe-49aa-9f79-41da164ffa08-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw\" (UID: \"abc6bc4f-b6fe-49aa-9f79-41da164ffa08\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw" Oct 30 21:37:02.165876 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:02.160381 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56xsr\" (UniqueName: \"kubernetes.io/projected/abc6bc4f-b6fe-49aa-9f79-41da164ffa08-kube-api-access-56xsr\") pod \"payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw\" (UID: \"abc6bc4f-b6fe-49aa-9f79-41da164ffa08\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw" Oct 30 21:37:02.165876 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:02.161553 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc6bc4f-b6fe-49aa-9f79-41da164ffa08-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw\" (UID: \"abc6bc4f-b6fe-49aa-9f79-41da164ffa08\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw" Oct 30 21:37:02.177743 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:02.177710 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56xsr\" (UniqueName: \"kubernetes.io/projected/abc6bc4f-b6fe-49aa-9f79-41da164ffa08-kube-api-access-56xsr\") pod \"payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw\" (UID: \"abc6bc4f-b6fe-49aa-9f79-41da164ffa08\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw" Oct 30 21:37:02.227211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:02.227177 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw" Oct 30 21:37:02.565496 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:02.565469 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw"] Oct 30 21:37:02.566484 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:37:02.566460 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc6bc4f_b6fe_49aa_9f79_41da164ffa08.slice/crio-5ed8747099fba5008b861ef66546e8ad60fb7bda61e62043a8f216e46b5b356a WatchSource:0}: Error finding container 5ed8747099fba5008b861ef66546e8ad60fb7bda61e62043a8f216e46b5b356a: Status 404 returned error can't find the container with id 5ed8747099fba5008b861ef66546e8ad60fb7bda61e62043a8f216e46b5b356a Oct 30 21:37:02.904439 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:02.904348 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:02.943559 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:02.943524 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79" event={"ID":"c7b2ea58-b940-4434-8b89-c3d7c0bc3cba","Type":"ContainerStarted","Data":"12de89c06f58497c1090ae71ea18c1f56245e65d5e1ab8ab7266403227676ca4"} Oct 30 21:37:02.943791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:02.943584 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79" event={"ID":"c7b2ea58-b940-4434-8b89-c3d7c0bc3cba","Type":"ContainerStarted","Data":"bf97457de0ac059e9debfd7ca7550726c96c066b3c547fdb708830bd400f2366"} Oct 30 21:37:02.943791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:02.943634 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79" Oct 30 21:37:02.945704 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:02.945679 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw" event={"ID":"abc6bc4f-b6fe-49aa-9f79-41da164ffa08","Type":"ContainerStarted","Data":"14157862f3c04e54eb7387cdfb2527aaf664321bcb8403c105ce4962fd1d228d"} Oct 30 21:37:02.945842 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:02.945717 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw" event={"ID":"abc6bc4f-b6fe-49aa-9f79-41da164ffa08","Type":"ContainerStarted","Data":"5ed8747099fba5008b861ef66546e8ad60fb7bda61e62043a8f216e46b5b356a"} Oct 30 21:37:02.945842 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:02.945784 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw" Oct 30 21:37:02.993044 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:02.992949 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79" podStartSLOduration=1.9929247060000002 podStartE2EDuration="1.992924706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:02.983607474 +0000 UTC m=+491.579424156" watchObservedRunningTime="2026-10-30 21:37:02.992924706 +0000 UTC m=+491.588741369" Oct 30 21:37:03.021406 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:03.021350 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw" podStartSLOduration=2.021335667 podStartE2EDuration="2.021335667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:03.020582127 +0000 UTC m=+491.616398807" watchObservedRunningTime="2026-10-30 21:37:03.021335667 +0000 UTC m=+491.617152327" Oct 30 21:37:03.082041 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:03.081997 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:03.887235 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:03.887178 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:03.988778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:03.986456 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:04.093800 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:04.093746 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:04.977419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:04.977383 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:05.079466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:05.079401 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:05.181313 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:05.181276 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:05.828011 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:05.827973 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:05.927526 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:05.927497 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:06.031995 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:06.031963 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:06.778626 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:06.778589 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:06.882320 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:06.882285 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:06.983088 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:06.983055 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:07.700802 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:07.683352 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:07.786410 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:07.786372 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:07.907425 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:07.907385 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:08.737545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:08.737512 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:09.718846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:09.718810 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:10.837038 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:10.836946 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:10.943943 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:10.943884 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:37:10.944127 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:10.943972 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" podUID="0f258bc9-d2fd-446a-9154-36caf951e081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:37:11.318601 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:11.318567 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:11.744587 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:11.744507 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:12.094774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:12.094089 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:12.198452 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:12.198416 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:12.324217 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:12.324185 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:12.572366 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:12.572329 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:12.848993 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:12.848897 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:13.152317 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:13.152098 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:13.284362 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:13.284293 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:13.428796 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:13.426922 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:13.872783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:13.864136 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb"] Oct 30 21:37:13.905786 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:13.904659 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb"] Oct 30 21:37:13.905786 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:13.904799 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:37:13.923971 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:13.923629 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0fa864\"/\"openshift-service-ca.crt\"" Oct 30 21:37:13.923971 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:13.923787 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0fa864\"/\"default-dockercfg-jldb5\"" Oct 30 21:37:13.923971 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:13.923906 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0fa864\"/\"simulator-w4-0fa864-kserve-self-signed-certs\"" Oct 30 21:37:13.927259 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:13.927187 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0fa864\"/\"kube-root-ca.crt\"" Oct 30 21:37:13.932334 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:13.932305 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:13.999899 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:13.998917 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79" Oct 30 21:37:14.001331 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:14.000782 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw" Oct 30 21:37:14.025025 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:14.024200 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-dshm\") pod \"simulator-w4-0fa864-kserve-657cf64664-d8hhb\" (UID: \"c2ea2669-ebc8-484c-aaef-33298655e79e\") " pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:37:14.025025 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:14.024284 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-home\") pod \"simulator-w4-0fa864-kserve-657cf64664-d8hhb\" (UID: \"c2ea2669-ebc8-484c-aaef-33298655e79e\") " pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:37:14.025025 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:14.024343 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-model-cache\") pod \"simulator-w4-0fa864-kserve-657cf64664-d8hhb\" (UID: \"c2ea2669-ebc8-484c-aaef-33298655e79e\") " pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:37:14.025025 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:14.024391 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-tmp-dir\") pod \"simulator-w4-0fa864-kserve-657cf64664-d8hhb\" (UID: \"c2ea2669-ebc8-484c-aaef-33298655e79e\") " pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:37:14.025025 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:14.024464 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr8wq\" (UniqueName: \"kubernetes.io/projected/c2ea2669-ebc8-484c-aaef-33298655e79e-kube-api-access-xr8wq\") pod \"simulator-w4-0fa864-kserve-657cf64664-d8hhb\" (UID: \"c2ea2669-ebc8-484c-aaef-33298655e79e\") " pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:37:14.025025 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:14.024517 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2ea2669-ebc8-484c-aaef-33298655e79e-tls-certs\") pod \"simulator-w4-0fa864-kserve-657cf64664-d8hhb\" (UID: \"c2ea2669-ebc8-484c-aaef-33298655e79e\") " pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:37:14.127781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:14.127676 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-dshm\") pod \"simulator-w4-0fa864-kserve-657cf64664-d8hhb\" (UID: \"c2ea2669-ebc8-484c-aaef-33298655e79e\") " pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:37:14.127781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:14.127719 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-home\") pod \"simulator-w4-0fa864-kserve-657cf64664-d8hhb\" (UID: \"c2ea2669-ebc8-484c-aaef-33298655e79e\") " pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:37:14.128021 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:14.127807 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-model-cache\") pod \"simulator-w4-0fa864-kserve-657cf64664-d8hhb\" (UID: \"c2ea2669-ebc8-484c-aaef-33298655e79e\") " pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:37:14.128021 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:14.127840 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-tmp-dir\") pod \"simulator-w4-0fa864-kserve-657cf64664-d8hhb\" (UID: \"c2ea2669-ebc8-484c-aaef-33298655e79e\") " pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:37:14.128021 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:14.127910 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr8wq\" (UniqueName: \"kubernetes.io/projected/c2ea2669-ebc8-484c-aaef-33298655e79e-kube-api-access-xr8wq\") pod \"simulator-w4-0fa864-kserve-657cf64664-d8hhb\" (UID: \"c2ea2669-ebc8-484c-aaef-33298655e79e\") " pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:37:14.128021 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:14.127958 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2ea2669-ebc8-484c-aaef-33298655e79e-tls-certs\") pod \"simulator-w4-0fa864-kserve-657cf64664-d8hhb\" (UID: \"c2ea2669-ebc8-484c-aaef-33298655e79e\") " pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:37:14.129434 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:14.129408 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-home\") pod \"simulator-w4-0fa864-kserve-657cf64664-d8hhb\" (UID: \"c2ea2669-ebc8-484c-aaef-33298655e79e\") " pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:37:14.129823 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:14.129775 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-tmp-dir\") pod \"simulator-w4-0fa864-kserve-657cf64664-d8hhb\" (UID: \"c2ea2669-ebc8-484c-aaef-33298655e79e\") " pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:37:14.130164 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:14.130125 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-model-cache\") pod \"simulator-w4-0fa864-kserve-657cf64664-d8hhb\" (UID: \"c2ea2669-ebc8-484c-aaef-33298655e79e\") " pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:37:14.134342 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:14.132821 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2ea2669-ebc8-484c-aaef-33298655e79e-tls-certs\") pod \"simulator-w4-0fa864-kserve-657cf64664-d8hhb\" (UID: \"c2ea2669-ebc8-484c-aaef-33298655e79e\") " pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:37:14.148855 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:14.136941 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-dshm\") pod \"simulator-w4-0fa864-kserve-657cf64664-d8hhb\" (UID: \"c2ea2669-ebc8-484c-aaef-33298655e79e\") " pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:37:14.187832 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:14.184628 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr8wq\" (UniqueName: \"kubernetes.io/projected/c2ea2669-ebc8-484c-aaef-33298655e79e-kube-api-access-xr8wq\") pod \"simulator-w4-0fa864-kserve-657cf64664-d8hhb\" (UID: \"c2ea2669-ebc8-484c-aaef-33298655e79e\") " pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:37:14.247548 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:14.243927 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:37:14.357819 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:14.351163 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:14.721844 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:14.720451 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb"] Oct 30 21:37:15.140793 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:15.139740 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" event={"ID":"c2ea2669-ebc8-484c-aaef-33298655e79e","Type":"ContainerStarted","Data":"c8b66342f364a4cac46948e5cc97c677dc7bc29f72405ecc6c56bbc648a8911c"} Oct 30 21:37:15.140793 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:15.139812 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" event={"ID":"c2ea2669-ebc8-484c-aaef-33298655e79e","Type":"ContainerStarted","Data":"8aed53f11cb7bcfcf75228fbe900d8cbf825764f63ee7e622af222411dd1a544"} Oct 30 21:37:15.177777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:15.176945 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podStartSLOduration=2.176922585 podStartE2EDuration="2.176922585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:15.175155195 +0000 UTC m=+503.770971856" watchObservedRunningTime="2026-10-30 21:37:15.176922585 +0000 UTC m=+503.772739250" Oct 30 21:37:15.246399 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:15.246364 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:37:15.371777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:15.367684 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:16.524376 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:16.512892 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:16.715140 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:16.714568 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:17.238285 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:17.238254 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:17.385774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:17.385207 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:17.621606 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:17.621244 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:17.733591 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:17.733499 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:18.198236 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:18.177103 5801 generic.go:356] "Generic (PLEG): container finished" podID="0f258bc9-d2fd-446a-9154-36caf951e081" containerID="76fe2b8ce6f3ea3a2d1c7ecec5e9b7bd17b1b9a550a56410f745684e91c3ce96" exitCode=0 Oct 30 21:37:18.198236 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:18.177240 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" event={"ID":"0f258bc9-d2fd-446a-9154-36caf951e081","Type":"ContainerDied","Data":"76fe2b8ce6f3ea3a2d1c7ecec5e9b7bd17b1b9a550a56410f745684e91c3ce96"} Oct 30 21:37:18.198236 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:18.177674 5801 scope.go:122] "RemoveContainer" containerID="76fe2b8ce6f3ea3a2d1c7ecec5e9b7bd17b1b9a550a56410f745684e91c3ce96" Oct 30 21:37:18.198236 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:18.179226 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:37:18.198236 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:18.179315 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:37:18.198236 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:18.179363 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:37:18.314866 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:18.311151 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:18.461591 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:18.461484 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:18.699883 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:18.699847 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:18.916491 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:18.916452 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:19.188831 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:19.188310 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" event={"ID":"0f258bc9-d2fd-446a-9154-36caf951e081","Type":"ContainerStarted","Data":"91184ee5cdc7e7d95c441873ef8ab2d8bbe6cf53bd147cd47d8682d422463155"} Oct 30 21:37:19.690123 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:19.690073 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:20.138378 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:20.138338 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:37:20.139258 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:20.139226 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 21:37:20.139376 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:20.139289 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" podUID="0f258bc9-d2fd-446a-9154-36caf951e081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 21:37:20.489218 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:20.489108 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:21.073673 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:21.073600 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:21.138099 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:21.138060 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 21:37:21.138288 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:21.138140 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" podUID="0f258bc9-d2fd-446a-9154-36caf951e081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 21:37:21.722479 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:21.453111 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:21.904961 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:21.904924 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:22.138621 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:22.138591 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 21:37:22.470445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:22.138641 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" podUID="0f258bc9-d2fd-446a-9154-36caf951e081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 21:37:22.470445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:22.209832 5801 generic.go:356] "Generic (PLEG): container finished" podID="8aa256c9-7e49-4144-9c43-91c8cdca8a7a" containerID="5eff4855aa7a6356c524849160e4af14d893fcf75713e393ff74f13aaffd3211" exitCode=0 Oct 30 21:37:22.470445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:22.209903 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" event={"ID":"8aa256c9-7e49-4144-9c43-91c8cdca8a7a","Type":"ContainerDied","Data":"5eff4855aa7a6356c524849160e4af14d893fcf75713e393ff74f13aaffd3211"} Oct 30 21:37:22.470445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:22.210332 5801 scope.go:122] "RemoveContainer" containerID="5eff4855aa7a6356c524849160e4af14d893fcf75713e393ff74f13aaffd3211" Oct 30 21:37:22.470445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:22.211740 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:37:22.470445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:22.211797 5801 generic.go:356] "Generic (PLEG): container finished" podID="9ea6c627-8885-490d-88ee-10befd58f673" containerID="26f388b059480b1f1ecc6f86cea4c809d8d4230370c30f25da41e897eca4993a" exitCode=0 Oct 30 21:37:22.470445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:22.211825 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:37:22.470445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:22.211857 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:37:22.470445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:22.211854 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" event={"ID":"9ea6c627-8885-490d-88ee-10befd58f673","Type":"ContainerDied","Data":"26f388b059480b1f1ecc6f86cea4c809d8d4230370c30f25da41e897eca4993a"} Oct 30 21:37:22.470445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:22.212331 5801 scope.go:122] "RemoveContainer" containerID="26f388b059480b1f1ecc6f86cea4c809d8d4230370c30f25da41e897eca4993a" Oct 30 21:37:22.470445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:22.213470 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:37:22.470445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:22.213526 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:37:22.470445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:22.213559 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:37:22.470445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:22.241534 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:22.578775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:22.578308 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:22.727946 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:22.727862 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:22.851494 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:22.851363 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:23.041874 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:23.034471 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:23.140873 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:23.140826 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 21:37:23.141078 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:23.140886 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" podUID="0f258bc9-d2fd-446a-9154-36caf951e081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 21:37:23.260832 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:23.255413 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" event={"ID":"9ea6c627-8885-490d-88ee-10befd58f673","Type":"ContainerStarted","Data":"a11e2faeda89f468c8c762c0506a20872a9c71771fa904f821982ddf4cf4501e"} Oct 30 21:37:23.300836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:23.297242 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" event={"ID":"8aa256c9-7e49-4144-9c43-91c8cdca8a7a","Type":"ContainerStarted","Data":"9386397b7b5ab358b2edd8b811251d9f1c14af3554fbebde08ccf1b9b75fce15"} Oct 30 21:37:23.444442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:23.444401 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 21:37:23.444716 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:23.444688 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" podUID="8aa256c9-7e49-4144-9c43-91c8cdca8a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 21:37:23.445082 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:23.445056 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:37:23.979837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:23.979228 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:37:23.979837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:23.979384 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 21:37:23.979837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:23.979427 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" podUID="9ea6c627-8885-490d-88ee-10befd58f673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 21:37:24.084869 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.083317 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:24.141365 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.138440 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 21:37:24.141365 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.138492 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" podUID="0f258bc9-d2fd-446a-9154-36caf951e081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 21:37:24.248395 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.248291 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:37:24.338444 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.338372 5801 generic.go:356] "Generic (PLEG): container finished" podID="ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" containerID="30c565eb326a7c2d89560e7c9e91fd808763b066617b64f4335aa365f6f103b5" exitCode=0 Oct 30 21:37:24.338649 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.338445 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" event={"ID":"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2","Type":"ContainerDied","Data":"30c565eb326a7c2d89560e7c9e91fd808763b066617b64f4335aa365f6f103b5"} Oct 30 21:37:24.338903 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.338862 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:37:24.338991 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.338932 5801 scope.go:122] "RemoveContainer" containerID="30c565eb326a7c2d89560e7c9e91fd808763b066617b64f4335aa365f6f103b5" Oct 30 21:37:24.340499 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.340466 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:37:24.340576 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.340557 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:37:24.340618 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.340610 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:37:24.341194 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.341115 5801 generic.go:356] "Generic (PLEG): container finished" podID="f205247c-2c3c-4dc3-a3d2-b46066918ae3" containerID="3e4a2b5342a6f8be4d987bb6df54b325b28bfe3f06f18725669bc5941da34a73" exitCode=0 Oct 30 21:37:24.341194 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.341180 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" event={"ID":"f205247c-2c3c-4dc3-a3d2-b46066918ae3","Type":"ContainerDied","Data":"3e4a2b5342a6f8be4d987bb6df54b325b28bfe3f06f18725669bc5941da34a73"} Oct 30 21:37:24.341819 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.341633 5801 scope.go:122] "RemoveContainer" containerID="3e4a2b5342a6f8be4d987bb6df54b325b28bfe3f06f18725669bc5941da34a73" Oct 30 21:37:24.343153 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.343000 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:37:24.343153 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.343072 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:37:24.343153 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.343117 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:37:24.458785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.449896 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:37:24.458785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.451898 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 21:37:24.458785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.451945 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" podUID="8aa256c9-7e49-4144-9c43-91c8cdca8a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 21:37:24.550415 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.550348 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:24.732815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.714963 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb"] Oct 30 21:37:24.732815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.726998 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:37:24.740982 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.740043 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-caa3dc\"/\"openshift-service-ca.crt\"" Oct 30 21:37:24.740982 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.740571 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-caa3dc\"/\"simulator-w5-caa3dc-kserve-self-signed-certs\"" Oct 30 21:37:24.740982 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.740656 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-caa3dc\"/\"kube-root-ca.crt\"" Oct 30 21:37:24.740982 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.740938 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-caa3dc\"/\"default-dockercfg-c2fpr\"" Oct 30 21:37:24.744601 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.744576 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb"] Oct 30 21:37:24.831777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.826971 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:24.835419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.834842 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-tls-certs\") pod \"simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb\" (UID: \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\") " pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:37:24.835419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.834902 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhdmd\" (UniqueName: \"kubernetes.io/projected/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-kube-api-access-dhdmd\") pod \"simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb\" (UID: \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\") " pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:37:24.835419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.834945 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-tmp-dir\") pod \"simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb\" (UID: \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\") " pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:37:24.835419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.834972 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-home\") pod \"simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb\" (UID: \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\") " pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:37:24.835419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.835029 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-model-cache\") pod \"simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb\" (UID: \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\") " pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:37:24.835419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.835078 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-dshm\") pod \"simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb\" (UID: \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\") " pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:37:24.908799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.903914 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:37:24.908799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.904079 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:37:24.941923 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.941882 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-model-cache\") pod \"simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb\" (UID: \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\") " pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:37:24.950781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.944199 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-model-cache\") pod \"simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb\" (UID: \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\") " pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:37:24.950997 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.950978 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-dshm\") pod \"simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb\" (UID: \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\") " pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:37:24.954950 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.954912 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-tls-certs\") pod \"simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb\" (UID: \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\") " pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:37:24.956919 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.955232 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhdmd\" (UniqueName: \"kubernetes.io/projected/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-kube-api-access-dhdmd\") pod \"simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb\" (UID: \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\") " pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:37:24.996429 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.957083 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-tmp-dir\") pod \"simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb\" (UID: \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\") " pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:37:24.997660 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.980452 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-tmp-dir\") pod \"simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb\" (UID: \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\") " pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:37:24.997660 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.997512 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-home\") pod \"simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb\" (UID: \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\") " pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:37:24.998743 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:24.998716 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-home\") pod \"simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb\" (UID: \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\") " pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:37:25.009880 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:25.009849 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 21:37:25.010000 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:25.009908 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" podUID="9ea6c627-8885-490d-88ee-10befd58f673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 21:37:25.025828 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:25.025553 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-tls-certs\") pod \"simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb\" (UID: \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\") " pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:37:25.028456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:25.028404 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhdmd\" (UniqueName: \"kubernetes.io/projected/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-kube-api-access-dhdmd\") pod \"simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb\" (UID: \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\") " pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:37:25.070775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:25.065065 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-dshm\") pod \"simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb\" (UID: \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\") " pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:37:25.141832 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:25.141176 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 21:37:25.141832 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:25.141241 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" podUID="0f258bc9-d2fd-446a-9154-36caf951e081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 21:37:25.147646 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:25.147390 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:37:25.357126 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:25.357093 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:25.484420 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:25.471439 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 21:37:25.484420 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:25.471594 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" podUID="8aa256c9-7e49-4144-9c43-91c8cdca8a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 21:37:25.665721 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:25.665568 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" event={"ID":"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2","Type":"ContainerStarted","Data":"cdfb2e6cbcd8969cd1624d0c0e07e081187aeb4b6ccb9b11f0ce73b91b9cc3db"} Oct 30 21:37:25.697822 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:25.695866 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:37:25.734826 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:25.705732 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 21:37:25.734826 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:25.705843 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" podUID="ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 21:37:25.734826 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:25.712508 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" event={"ID":"f205247c-2c3c-4dc3-a3d2-b46066918ae3","Type":"ContainerStarted","Data":"a65c88cc36092db46b9a2e5e65de11a0cc90ba9c4bb08da88864c9d85da15eb7"} Oct 30 21:37:25.905169 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:25.905099 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:37:25.942912 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:25.925871 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb"] Oct 30 21:37:25.942912 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:25.925923 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:37:25.983587 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:25.983152 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 21:37:25.983587 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:25.983214 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" podUID="9ea6c627-8885-490d-88ee-10befd58f673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 21:37:26.157464 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:26.156069 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:37:26.167413 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:26.165053 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:37:26.387395 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:26.387045 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:37:26.388134 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:26.387765 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 30 21:37:26.388134 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:26.387830 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" podUID="f205247c-2c3c-4dc3-a3d2-b46066918ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 30 21:37:26.402901 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:26.388349 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:26.451777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:26.451181 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 21:37:26.451777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:26.451260 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" podUID="8aa256c9-7e49-4144-9c43-91c8cdca8a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 21:37:26.682361 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:26.682291 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 21:37:26.682652 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:26.682597 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" podUID="ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 21:37:26.778204 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:26.778167 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" event={"ID":"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1","Type":"ContainerStarted","Data":"d76cd2f3ed15ecd4586e894a1b5013e8cd0f828d9f99392074c76c24f952ce7e"} Oct 30 21:37:26.909960 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:26.909906 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:37:26.987399 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:26.987293 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 21:37:26.987679 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:26.987637 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" podUID="9ea6c627-8885-490d-88ee-10befd58f673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 21:37:27.215378 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:27.215245 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:27.308239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:27.308204 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 30 21:37:27.308506 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:27.308479 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" podUID="f205247c-2c3c-4dc3-a3d2-b46066918ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 30 21:37:27.444778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:27.441030 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 21:37:27.444778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:27.441092 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" podUID="8aa256c9-7e49-4144-9c43-91c8cdca8a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 21:37:27.683198 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:27.683112 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 21:37:27.683459 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:27.683426 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" podUID="ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 21:37:27.805306 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:27.805268 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" event={"ID":"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1","Type":"ContainerStarted","Data":"b7e7d3188cd510e938bf4835623a2034161095aa47f6b6e80f55a65db60776c4"} Oct 30 21:37:27.867779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:27.843142 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podStartSLOduration=3.843118454 podStartE2EDuration="3.843118454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:27.833141465 +0000 UTC m=+516.428958142" watchObservedRunningTime="2026-10-30 21:37:27.843118454 +0000 UTC m=+516.438935115" Oct 30 21:37:27.981216 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:27.981120 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 21:37:27.981462 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:27.981429 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" podUID="9ea6c627-8885-490d-88ee-10befd58f673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 21:37:28.038897 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:28.038860 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:28.149906 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:28.149875 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:37:28.181792 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:28.181693 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:28.305789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:28.305317 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 30 21:37:28.305789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:28.305378 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" podUID="f205247c-2c3c-4dc3-a3d2-b46066918ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 30 21:37:28.495941 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:28.470826 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 21:37:28.495941 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:28.470890 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" podUID="8aa256c9-7e49-4144-9c43-91c8cdca8a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 21:37:28.537451 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:28.536917 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:37:28.694313 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:28.694227 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 21:37:28.713531 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:28.713484 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" podUID="ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 21:37:28.767487 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:28.767384 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:28.978350 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:28.978249 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 21:37:28.978631 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:28.978587 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" podUID="9ea6c627-8885-490d-88ee-10befd58f673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 21:37:29.306348 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.306310 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 30 21:37:29.306976 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.306927 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" podUID="f205247c-2c3c-4dc3-a3d2-b46066918ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 30 21:37:29.442229 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.441969 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:37:29.443830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.443641 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:37:29.606974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.606530 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2"] Oct 30 21:37:29.629473 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.629317 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:29.630347 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.629840 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2"] Oct 30 21:37:29.630538 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.630306 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:37:29.635811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.634829 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7dfff0\"/\"simulator-w6-7dfff0-kserve-self-signed-certs\"" Oct 30 21:37:29.635811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.635034 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7dfff0\"/\"default-dockercfg-xldhw\"" Oct 30 21:37:29.635811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.635640 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7dfff0\"/\"kube-root-ca.crt\"" Oct 30 21:37:29.635811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.635707 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7dfff0\"/\"openshift-service-ca.crt\"" Oct 30 21:37:29.728816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.709846 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 21:37:29.728816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.709905 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" podUID="ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 21:37:29.728816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.711527 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cqbd\" (UniqueName: \"kubernetes.io/projected/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-kube-api-access-2cqbd\") pod \"simulator-w6-7dfff0-kserve-86b668d6b-6g2c2\" (UID: \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\") " pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:37:29.728816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.711628 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-model-cache\") pod \"simulator-w6-7dfff0-kserve-86b668d6b-6g2c2\" (UID: \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\") " pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:37:29.728816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.711686 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-tls-certs\") pod \"simulator-w6-7dfff0-kserve-86b668d6b-6g2c2\" (UID: \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\") " pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:37:29.728816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.711719 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-dshm\") pod \"simulator-w6-7dfff0-kserve-86b668d6b-6g2c2\" (UID: \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\") " pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:37:29.728816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.711788 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-home\") pod \"simulator-w6-7dfff0-kserve-86b668d6b-6g2c2\" (UID: \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\") " pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:37:29.728816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.711850 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-tmp-dir\") pod \"simulator-w6-7dfff0-kserve-86b668d6b-6g2c2\" (UID: \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\") " pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:37:29.737918 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.737874 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:29.853812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.831552 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-home\") pod \"simulator-w6-7dfff0-kserve-86b668d6b-6g2c2\" (UID: \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\") " pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:37:29.853812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.831629 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-tmp-dir\") pod \"simulator-w6-7dfff0-kserve-86b668d6b-6g2c2\" (UID: \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\") " pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:37:29.853812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.831691 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cqbd\" (UniqueName: \"kubernetes.io/projected/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-kube-api-access-2cqbd\") pod \"simulator-w6-7dfff0-kserve-86b668d6b-6g2c2\" (UID: \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\") " pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:37:29.853812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.831801 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-model-cache\") pod \"simulator-w6-7dfff0-kserve-86b668d6b-6g2c2\" (UID: \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\") " pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:37:29.853812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.831865 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-tls-certs\") pod \"simulator-w6-7dfff0-kserve-86b668d6b-6g2c2\" (UID: \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\") " pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:37:29.853812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.831901 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-dshm\") pod \"simulator-w6-7dfff0-kserve-86b668d6b-6g2c2\" (UID: \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\") " pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:37:29.853812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.852510 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-home\") pod \"simulator-w6-7dfff0-kserve-86b668d6b-6g2c2\" (UID: \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\") " pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:37:29.853812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.852909 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-tmp-dir\") pod \"simulator-w6-7dfff0-kserve-86b668d6b-6g2c2\" (UID: \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\") " pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:37:29.853812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.853151 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-model-cache\") pod \"simulator-w6-7dfff0-kserve-86b668d6b-6g2c2\" (UID: \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\") " pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:37:29.902858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.880874 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-dshm\") pod \"simulator-w6-7dfff0-kserve-86b668d6b-6g2c2\" (UID: \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\") " pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:37:29.925590 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.922449 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cqbd\" (UniqueName: \"kubernetes.io/projected/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-kube-api-access-2cqbd\") pod \"simulator-w6-7dfff0-kserve-86b668d6b-6g2c2\" (UID: \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\") " pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:37:29.932423 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:29.932387 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-tls-certs\") pod \"simulator-w6-7dfff0-kserve-86b668d6b-6g2c2\" (UID: \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\") " pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:37:30.004848 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:30.004070 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:37:30.006430 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:30.006343 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:37:30.031495 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:30.031284 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:37:30.306820 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:30.306456 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 30 21:37:30.306820 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:30.306512 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" podUID="f205247c-2c3c-4dc3-a3d2-b46066918ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 30 21:37:30.328131 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:37:30.328091 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:37:30.405945 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:30.405914 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2"] Oct 30 21:37:30.407738 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:37:30.407708 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod986c5f2e_ccbb_4d9f_9773_e83e7de17fff.slice/crio-32ebeabc1d0391046f9079c4f9b5e2a5e334e0281aad200da2befa9481c93782 WatchSource:0}: Error finding container 32ebeabc1d0391046f9079c4f9b5e2a5e334e0281aad200da2befa9481c93782: Status 404 returned error can't find the container with id 32ebeabc1d0391046f9079c4f9b5e2a5e334e0281aad200da2befa9481c93782 Oct 30 21:37:30.439179 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:30.439140 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:30.691831 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:30.681116 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 21:37:30.691831 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:30.681177 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" podUID="ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 21:37:30.923659 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:30.923620 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" event={"ID":"986c5f2e-ccbb-4d9f-9773-e83e7de17fff","Type":"ContainerStarted","Data":"78761f0cd4b28ffd158c5476b75a143d7d8bfed44c4cfb3b7d175d1198ef3e82"} Oct 30 21:37:30.923828 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:30.923671 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" event={"ID":"986c5f2e-ccbb-4d9f-9773-e83e7de17fff","Type":"ContainerStarted","Data":"32ebeabc1d0391046f9079c4f9b5e2a5e334e0281aad200da2befa9481c93782"} Oct 30 21:37:30.947279 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:30.947198 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podStartSLOduration=1.9471862720000002 podStartE2EDuration="1.947186272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:30.945676593 +0000 UTC m=+519.541493256" watchObservedRunningTime="2026-10-30 21:37:30.947186272 +0000 UTC m=+519.543002933" Oct 30 21:37:31.032310 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:31.032278 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:37:31.148472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:31.148431 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:31.303313 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:31.303277 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 30 21:37:31.303490 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:31.303344 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" podUID="f205247c-2c3c-4dc3-a3d2-b46066918ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 30 21:37:31.681833 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:31.681204 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 21:37:31.681833 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:31.681281 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" podUID="ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 21:37:31.838509 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:31.838475 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:32.307390 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:32.307364 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:37:32.307684 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:32.307652 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:37:32.308230 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:32.308212 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:37:32.577480 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:32.577390 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:32.685834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:32.685802 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:37:32.686865 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:32.686846 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:37:33.413817 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:33.413779 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:33.511927 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:33.511894 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:34.258180 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:34.258142 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:35.148542 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:35.148510 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:37:35.161884 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:35.161857 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:37:35.240109 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:35.240072 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:35.945403 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:35.945370 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:35.966056 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:35.966027 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:37:36.649969 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:36.649938 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:37.206273 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.206233 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5"] Oct 30 21:37:37.209861 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.209841 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:37:37.219520 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.219494 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1d8034\"/\"kube-root-ca.crt\"" Oct 30 21:37:37.219938 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.219638 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1d8034\"/\"simulator-w3-1d8034-kserve-self-signed-certs\"" Oct 30 21:37:37.219938 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.219661 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1d8034\"/\"default-dockercfg-n6qtb\"" Oct 30 21:37:37.219938 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.219667 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1d8034\"/\"openshift-service-ca.crt\"" Oct 30 21:37:37.221108 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.221091 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5"] Oct 30 21:37:37.245635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.245609 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:37.318032 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.318008 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-tmp-dir\") pod \"simulator-w3-1d8034-kserve-8648d847b4-g2qr5\" (UID: \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\") " pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:37:37.318152 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.318042 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-home\") pod \"simulator-w3-1d8034-kserve-8648d847b4-g2qr5\" (UID: \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\") " pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:37:37.318152 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.318070 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-dshm\") pod \"simulator-w3-1d8034-kserve-8648d847b4-g2qr5\" (UID: \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\") " pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:37:37.318152 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.318109 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-tls-certs\") pod \"simulator-w3-1d8034-kserve-8648d847b4-g2qr5\" (UID: \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\") " pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:37:37.318268 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.318188 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-model-cache\") pod \"simulator-w3-1d8034-kserve-8648d847b4-g2qr5\" (UID: \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\") " pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:37:37.318268 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.318225 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrldb\" (UniqueName: \"kubernetes.io/projected/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-kube-api-access-lrldb\") pod \"simulator-w3-1d8034-kserve-8648d847b4-g2qr5\" (UID: \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\") " pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:37:37.419919 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.419874 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-tmp-dir\") pod \"simulator-w3-1d8034-kserve-8648d847b4-g2qr5\" (UID: \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\") " pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:37:37.419919 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.419919 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-home\") pod \"simulator-w3-1d8034-kserve-8648d847b4-g2qr5\" (UID: \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\") " pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:37:37.420177 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.419950 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-dshm\") pod \"simulator-w3-1d8034-kserve-8648d847b4-g2qr5\" (UID: \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\") " pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:37:37.420177 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.419965 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-tls-certs\") pod \"simulator-w3-1d8034-kserve-8648d847b4-g2qr5\" (UID: \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\") " pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:37:37.420177 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.419997 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-model-cache\") pod \"simulator-w3-1d8034-kserve-8648d847b4-g2qr5\" (UID: \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\") " pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:37:37.420177 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.420026 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrldb\" (UniqueName: \"kubernetes.io/projected/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-kube-api-access-lrldb\") pod \"simulator-w3-1d8034-kserve-8648d847b4-g2qr5\" (UID: \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\") " pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:37:37.420409 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.420289 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-tmp-dir\") pod \"simulator-w3-1d8034-kserve-8648d847b4-g2qr5\" (UID: \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\") " pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:37:37.420409 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.420360 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-home\") pod \"simulator-w3-1d8034-kserve-8648d847b4-g2qr5\" (UID: \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\") " pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:37:37.420621 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.420603 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-model-cache\") pod \"simulator-w3-1d8034-kserve-8648d847b4-g2qr5\" (UID: \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\") " pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:37:37.423369 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.423342 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-dshm\") pod \"simulator-w3-1d8034-kserve-8648d847b4-g2qr5\" (UID: \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\") " pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:37:37.423839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.423819 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-tls-certs\") pod \"simulator-w3-1d8034-kserve-8648d847b4-g2qr5\" (UID: \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\") " pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:37:37.437679 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.437648 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrldb\" (UniqueName: \"kubernetes.io/projected/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-kube-api-access-lrldb\") pod \"simulator-w3-1d8034-kserve-8648d847b4-g2qr5\" (UID: \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\") " pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:37:37.445003 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.444904 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:37.524290 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.524209 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:37:37.800209 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.800183 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5"] Oct 30 21:37:37.802741 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:37:37.802704 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod363ebad1_96ba_4cf4_b0b9_2a93cbf1bbdb.slice/crio-2cda24714eefab85122abf3c79f9c6e49b7692bba2378421b399bf1a1057cd51 WatchSource:0}: Error finding container 2cda24714eefab85122abf3c79f9c6e49b7692bba2378421b399bf1a1057cd51: Status 404 returned error can't find the container with id 2cda24714eefab85122abf3c79f9c6e49b7692bba2378421b399bf1a1057cd51 Oct 30 21:37:37.968288 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.968244 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" event={"ID":"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb","Type":"ContainerStarted","Data":"2e87a4e5ddf07a6d002b6aba59757416362357cbc088b588938d6511206d4c1d"} Oct 30 21:37:37.968288 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.968289 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" event={"ID":"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb","Type":"ContainerStarted","Data":"2cda24714eefab85122abf3c79f9c6e49b7692bba2378421b399bf1a1057cd51"} Oct 30 21:37:37.989442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:37.989392 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podStartSLOduration=0.989380388 podStartE2EDuration="989.380388ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:37.987508934 +0000 UTC m=+526.583325606" watchObservedRunningTime="2026-10-30 21:37:37.989380388 +0000 UTC m=+526.585197094" Oct 30 21:37:38.381980 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:38.375419 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:38.524663 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:38.524627 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:37:39.133851 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:39.133799 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:39.770208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:39.770170 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:40.032333 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:40.032260 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:37:40.045053 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:40.045027 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:37:40.142442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:40.142409 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:41.010489 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:41.010455 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:37:41.167545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:41.167489 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:41.962368 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:41.962332 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:42.756748 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:42.756713 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:43.568339 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:43.568307 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:43.670183 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:43.670151 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:44.538441 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:44.538403 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:45.442389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:45.442354 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:46.141133 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:46.141096 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:46.650304 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:46.650269 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:46.751776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:46.751719 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:46.945660 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:46.945571 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:47.524404 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:47.524373 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:37:47.536834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:47.536810 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:37:47.753900 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:47.753869 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:48.041583 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:48.041554 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:37:48.668517 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:48.668477 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:49.528979 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:49.528944 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:50.232992 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:50.232955 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:50.916422 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:50.916385 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p"] Oct 30 21:37:50.920930 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:50.920900 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:37:50.932698 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:50.932677 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-caa3dc\"/\"premium-w5-caa3dc-kserve-self-signed-certs\"" Oct 30 21:37:50.938747 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:50.936197 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p"] Oct 30 21:37:50.973483 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:50.973446 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:51.060127 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:51.060097 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-home\") pod \"premium-w5-caa3dc-kserve-745c5bcb58-d274p\" (UID: \"d49a00ed-7af8-4999-91d7-ee8174080727\") " pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:37:51.060283 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:51.060156 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-tmp-dir\") pod \"premium-w5-caa3dc-kserve-745c5bcb58-d274p\" (UID: \"d49a00ed-7af8-4999-91d7-ee8174080727\") " pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:37:51.060283 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:51.060243 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-model-cache\") pod \"premium-w5-caa3dc-kserve-745c5bcb58-d274p\" (UID: \"d49a00ed-7af8-4999-91d7-ee8174080727\") " pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:37:51.060365 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:51.060281 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-dshm\") pod \"premium-w5-caa3dc-kserve-745c5bcb58-d274p\" (UID: \"d49a00ed-7af8-4999-91d7-ee8174080727\") " pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:37:51.060365 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:51.060362 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d49a00ed-7af8-4999-91d7-ee8174080727-tls-certs\") pod \"premium-w5-caa3dc-kserve-745c5bcb58-d274p\" (UID: \"d49a00ed-7af8-4999-91d7-ee8174080727\") " pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:37:51.060440 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:51.060425 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzwf6\" (UniqueName: \"kubernetes.io/projected/d49a00ed-7af8-4999-91d7-ee8174080727-kube-api-access-dzwf6\") pod \"premium-w5-caa3dc-kserve-745c5bcb58-d274p\" (UID: \"d49a00ed-7af8-4999-91d7-ee8174080727\") " pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:37:51.076315 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:51.076270 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:51.161708 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:51.161673 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-model-cache\") pod \"premium-w5-caa3dc-kserve-745c5bcb58-d274p\" (UID: \"d49a00ed-7af8-4999-91d7-ee8174080727\") " pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:37:51.161908 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:51.161719 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-dshm\") pod \"premium-w5-caa3dc-kserve-745c5bcb58-d274p\" (UID: \"d49a00ed-7af8-4999-91d7-ee8174080727\") " pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:37:51.161908 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:51.161813 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d49a00ed-7af8-4999-91d7-ee8174080727-tls-certs\") pod \"premium-w5-caa3dc-kserve-745c5bcb58-d274p\" (UID: \"d49a00ed-7af8-4999-91d7-ee8174080727\") " pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:37:51.161908 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:51.161845 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzwf6\" (UniqueName: \"kubernetes.io/projected/d49a00ed-7af8-4999-91d7-ee8174080727-kube-api-access-dzwf6\") pod \"premium-w5-caa3dc-kserve-745c5bcb58-d274p\" (UID: \"d49a00ed-7af8-4999-91d7-ee8174080727\") " pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:37:51.161908 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:51.161885 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-home\") pod \"premium-w5-caa3dc-kserve-745c5bcb58-d274p\" (UID: \"d49a00ed-7af8-4999-91d7-ee8174080727\") " pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:37:51.162149 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:51.161950 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-tmp-dir\") pod \"premium-w5-caa3dc-kserve-745c5bcb58-d274p\" (UID: \"d49a00ed-7af8-4999-91d7-ee8174080727\") " pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:37:51.162239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:51.162215 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-model-cache\") pod \"premium-w5-caa3dc-kserve-745c5bcb58-d274p\" (UID: \"d49a00ed-7af8-4999-91d7-ee8174080727\") " pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:37:51.162300 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:51.162243 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-home\") pod \"premium-w5-caa3dc-kserve-745c5bcb58-d274p\" (UID: \"d49a00ed-7af8-4999-91d7-ee8174080727\") " pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:37:51.162339 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:51.162291 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-tmp-dir\") pod \"premium-w5-caa3dc-kserve-745c5bcb58-d274p\" (UID: \"d49a00ed-7af8-4999-91d7-ee8174080727\") " pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:37:51.163976 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:51.163957 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-dshm\") pod \"premium-w5-caa3dc-kserve-745c5bcb58-d274p\" (UID: \"d49a00ed-7af8-4999-91d7-ee8174080727\") " pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:37:51.164415 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:51.164398 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d49a00ed-7af8-4999-91d7-ee8174080727-tls-certs\") pod \"premium-w5-caa3dc-kserve-745c5bcb58-d274p\" (UID: \"d49a00ed-7af8-4999-91d7-ee8174080727\") " pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:37:51.174853 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:51.174802 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzwf6\" (UniqueName: \"kubernetes.io/projected/d49a00ed-7af8-4999-91d7-ee8174080727-kube-api-access-dzwf6\") pod \"premium-w5-caa3dc-kserve-745c5bcb58-d274p\" (UID: \"d49a00ed-7af8-4999-91d7-ee8174080727\") " pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:37:51.241143 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:51.241107 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:37:51.521307 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:37:51.521266 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd49a00ed_7af8_4999_91d7_ee8174080727.slice/crio-3bdf5e0a11abac49a964863adef928070331c8112b67a08469bfdcf93c4a1f0b WatchSource:0}: Error finding container 3bdf5e0a11abac49a964863adef928070331c8112b67a08469bfdcf93c4a1f0b: Status 404 returned error can't find the container with id 3bdf5e0a11abac49a964863adef928070331c8112b67a08469bfdcf93c4a1f0b Oct 30 21:37:51.525776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:51.525739 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p"] Oct 30 21:37:51.758803 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:51.758739 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:52.054087 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:52.054050 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" event={"ID":"d49a00ed-7af8-4999-91d7-ee8174080727","Type":"ContainerStarted","Data":"fe597c9e61aa17655ab23710562b507ead65ae66da9d138535e51044324e8375"} Oct 30 21:37:52.054244 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:52.054099 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" event={"ID":"d49a00ed-7af8-4999-91d7-ee8174080727","Type":"ContainerStarted","Data":"3bdf5e0a11abac49a964863adef928070331c8112b67a08469bfdcf93c4a1f0b"} Oct 30 21:37:52.079416 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:52.079365 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podStartSLOduration=2.079343005 podStartE2EDuration="2.079343005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:52.077648444 +0000 UTC m=+540.673465105" watchObservedRunningTime="2026-10-30 21:37:52.079343005 +0000 UTC m=+540.675159664" Oct 30 21:37:52.241230 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:52.241185 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:37:52.558550 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:52.558517 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:53.341478 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:53.341445 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:53.446773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:53.446718 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:54.252336 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:54.252298 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:55.043279 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:55.043244 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:55.752077 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:55.752042 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:56.541229 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:56.541164 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:56.991182 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:56.991148 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:57.239561 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:57.239526 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:57.942504 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:57.942471 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n"] Oct 30 21:37:57.945833 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:57.945801 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:57.946984 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:57.946965 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:37:57.955065 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:57.955039 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n"] Oct 30 21:37:57.955474 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:57.955453 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7dfff0\"/\"premium-w6-7dfff0-kserve-self-signed-certs\"" Oct 30 21:37:58.033139 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:58.033113 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-tmp-dir\") pod \"premium-w6-7dfff0-kserve-bc6bc9747-t8g7n\" (UID: \"2f352e82-4be6-4c2c-bda0-bc109f65b739\") " pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:37:58.033280 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:58.033153 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-model-cache\") pod \"premium-w6-7dfff0-kserve-bc6bc9747-t8g7n\" (UID: \"2f352e82-4be6-4c2c-bda0-bc109f65b739\") " pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:37:58.033280 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:58.033224 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxx4b\" (UniqueName: \"kubernetes.io/projected/2f352e82-4be6-4c2c-bda0-bc109f65b739-kube-api-access-bxx4b\") pod \"premium-w6-7dfff0-kserve-bc6bc9747-t8g7n\" (UID: \"2f352e82-4be6-4c2c-bda0-bc109f65b739\") " pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:37:58.033366 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:58.033281 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f352e82-4be6-4c2c-bda0-bc109f65b739-tls-certs\") pod \"premium-w6-7dfff0-kserve-bc6bc9747-t8g7n\" (UID: \"2f352e82-4be6-4c2c-bda0-bc109f65b739\") " pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:37:58.033408 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:58.033376 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-home\") pod \"premium-w6-7dfff0-kserve-bc6bc9747-t8g7n\" (UID: \"2f352e82-4be6-4c2c-bda0-bc109f65b739\") " pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:37:58.033478 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:58.033463 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-dshm\") pod \"premium-w6-7dfff0-kserve-bc6bc9747-t8g7n\" (UID: \"2f352e82-4be6-4c2c-bda0-bc109f65b739\") " pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:37:58.042275 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:58.042247 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:58.134187 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:58.134155 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-tmp-dir\") pod \"premium-w6-7dfff0-kserve-bc6bc9747-t8g7n\" (UID: \"2f352e82-4be6-4c2c-bda0-bc109f65b739\") " pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:37:58.134398 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:58.134201 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-model-cache\") pod \"premium-w6-7dfff0-kserve-bc6bc9747-t8g7n\" (UID: \"2f352e82-4be6-4c2c-bda0-bc109f65b739\") " pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:37:58.134398 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:58.134330 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxx4b\" (UniqueName: \"kubernetes.io/projected/2f352e82-4be6-4c2c-bda0-bc109f65b739-kube-api-access-bxx4b\") pod \"premium-w6-7dfff0-kserve-bc6bc9747-t8g7n\" (UID: \"2f352e82-4be6-4c2c-bda0-bc109f65b739\") " pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:37:58.134524 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:58.134427 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f352e82-4be6-4c2c-bda0-bc109f65b739-tls-certs\") pod \"premium-w6-7dfff0-kserve-bc6bc9747-t8g7n\" (UID: \"2f352e82-4be6-4c2c-bda0-bc109f65b739\") " pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:37:58.134524 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:58.134462 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-home\") pod \"premium-w6-7dfff0-kserve-bc6bc9747-t8g7n\" (UID: \"2f352e82-4be6-4c2c-bda0-bc109f65b739\") " pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:37:58.134524 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:58.134515 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-dshm\") pod \"premium-w6-7dfff0-kserve-bc6bc9747-t8g7n\" (UID: \"2f352e82-4be6-4c2c-bda0-bc109f65b739\") " pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:37:58.134699 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:58.134607 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-model-cache\") pod \"premium-w6-7dfff0-kserve-bc6bc9747-t8g7n\" (UID: \"2f352e82-4be6-4c2c-bda0-bc109f65b739\") " pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:37:58.134699 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:58.134667 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-tmp-dir\") pod \"premium-w6-7dfff0-kserve-bc6bc9747-t8g7n\" (UID: \"2f352e82-4be6-4c2c-bda0-bc109f65b739\") " pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:37:58.134859 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:58.134840 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-home\") pod \"premium-w6-7dfff0-kserve-bc6bc9747-t8g7n\" (UID: \"2f352e82-4be6-4c2c-bda0-bc109f65b739\") " pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:37:58.136886 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:58.136866 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-dshm\") pod \"premium-w6-7dfff0-kserve-bc6bc9747-t8g7n\" (UID: \"2f352e82-4be6-4c2c-bda0-bc109f65b739\") " pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:37:58.137077 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:58.137061 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f352e82-4be6-4c2c-bda0-bc109f65b739-tls-certs\") pod \"premium-w6-7dfff0-kserve-bc6bc9747-t8g7n\" (UID: \"2f352e82-4be6-4c2c-bda0-bc109f65b739\") " pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:37:58.148241 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:58.148212 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxx4b\" (UniqueName: \"kubernetes.io/projected/2f352e82-4be6-4c2c-bda0-bc109f65b739-kube-api-access-bxx4b\") pod \"premium-w6-7dfff0-kserve-bc6bc9747-t8g7n\" (UID: \"2f352e82-4be6-4c2c-bda0-bc109f65b739\") " pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:37:58.261815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:58.261703 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:37:58.537771 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:58.537724 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n"] Oct 30 21:37:58.539682 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:37:58.539651 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f352e82_4be6_4c2c_bda0_bc109f65b739.slice/crio-c9702f18e79b51f9eaad13016ec979b4538b3005c34ba55dbff24eec063fbaf4 WatchSource:0}: Error finding container c9702f18e79b51f9eaad13016ec979b4538b3005c34ba55dbff24eec063fbaf4: Status 404 returned error can't find the container with id c9702f18e79b51f9eaad13016ec979b4538b3005c34ba55dbff24eec063fbaf4 Oct 30 21:37:58.648802 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:58.648741 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:59.099355 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:59.099314 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" event={"ID":"2f352e82-4be6-4c2c-bda0-bc109f65b739","Type":"ContainerStarted","Data":"332ba3923ba97d6d3b413bf08d3f7d3e5f88bac7a64dccb08426f780e8052e6b"} Oct 30 21:37:59.099799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:59.099365 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" event={"ID":"2f352e82-4be6-4c2c-bda0-bc109f65b739","Type":"ContainerStarted","Data":"c9702f18e79b51f9eaad13016ec979b4538b3005c34ba55dbff24eec063fbaf4"} Oct 30 21:37:59.120167 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:59.120121 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podStartSLOduration=2.120108234 podStartE2EDuration="2.120108234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:59.118260141 +0000 UTC m=+547.714076820" watchObservedRunningTime="2026-10-30 21:37:59.120108234 +0000 UTC m=+547.715924894" Oct 30 21:37:59.262809 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:59.262776 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:37:59.435390 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:59.435314 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:59.539599 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:37:59.539561 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:00.238114 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:00.238073 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:00.277520 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:38:00.277483 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:38:01.044112 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.044072 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:01.241713 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.241677 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:38:01.254252 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.254230 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:38:01.660232 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.660196 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842"] Oct 30 21:38:01.664156 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.664123 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:38:01.667505 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.667484 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842"] Oct 30 21:38:01.670212 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.670190 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1a81ae\"/\"default-dockercfg-6lxnt\"" Oct 30 21:38:01.673027 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.672838 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1a81ae\"/\"premium-w1-1a81ae-kserve-self-signed-certs\"" Oct 30 21:38:01.673027 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.672945 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1a81ae\"/\"kube-root-ca.crt\"" Oct 30 21:38:01.716888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.716861 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:01.772708 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.772687 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hrg6\" (UniqueName: \"kubernetes.io/projected/d40a938f-1aa7-4897-b5a7-91257132baf7-kube-api-access-6hrg6\") pod \"premium-w1-1a81ae-kserve-679cdd99b8-mf842\" (UID: \"d40a938f-1aa7-4897-b5a7-91257132baf7\") " pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:38:01.772858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.772742 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d40a938f-1aa7-4897-b5a7-91257132baf7-tls-certs\") pod \"premium-w1-1a81ae-kserve-679cdd99b8-mf842\" (UID: \"d40a938f-1aa7-4897-b5a7-91257132baf7\") " pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:38:01.772858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.772845 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-tmp-dir\") pod \"premium-w1-1a81ae-kserve-679cdd99b8-mf842\" (UID: \"d40a938f-1aa7-4897-b5a7-91257132baf7\") " pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:38:01.772952 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.772920 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-home\") pod \"premium-w1-1a81ae-kserve-679cdd99b8-mf842\" (UID: \"d40a938f-1aa7-4897-b5a7-91257132baf7\") " pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:38:01.773063 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.773045 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-model-cache\") pod \"premium-w1-1a81ae-kserve-679cdd99b8-mf842\" (UID: \"d40a938f-1aa7-4897-b5a7-91257132baf7\") " pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:38:01.773111 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.773100 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-dshm\") pod \"premium-w1-1a81ae-kserve-679cdd99b8-mf842\" (UID: \"d40a938f-1aa7-4897-b5a7-91257132baf7\") " pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:38:01.773663 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.773649 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1a81ae\"/\"openshift-service-ca.crt\"" Oct 30 21:38:01.874323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.874290 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-home\") pod \"premium-w1-1a81ae-kserve-679cdd99b8-mf842\" (UID: \"d40a938f-1aa7-4897-b5a7-91257132baf7\") " pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:38:01.874523 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.874387 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-model-cache\") pod \"premium-w1-1a81ae-kserve-679cdd99b8-mf842\" (UID: \"d40a938f-1aa7-4897-b5a7-91257132baf7\") " pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:38:01.874523 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.874449 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-dshm\") pod \"premium-w1-1a81ae-kserve-679cdd99b8-mf842\" (UID: \"d40a938f-1aa7-4897-b5a7-91257132baf7\") " pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:38:01.874523 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.874480 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hrg6\" (UniqueName: \"kubernetes.io/projected/d40a938f-1aa7-4897-b5a7-91257132baf7-kube-api-access-6hrg6\") pod \"premium-w1-1a81ae-kserve-679cdd99b8-mf842\" (UID: \"d40a938f-1aa7-4897-b5a7-91257132baf7\") " pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:38:01.874697 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.874529 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d40a938f-1aa7-4897-b5a7-91257132baf7-tls-certs\") pod \"premium-w1-1a81ae-kserve-679cdd99b8-mf842\" (UID: \"d40a938f-1aa7-4897-b5a7-91257132baf7\") " pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:38:01.874697 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.874564 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-tmp-dir\") pod \"premium-w1-1a81ae-kserve-679cdd99b8-mf842\" (UID: \"d40a938f-1aa7-4897-b5a7-91257132baf7\") " pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:38:01.874851 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.874828 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-home\") pod \"premium-w1-1a81ae-kserve-679cdd99b8-mf842\" (UID: \"d40a938f-1aa7-4897-b5a7-91257132baf7\") " pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:38:01.874912 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.874863 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-model-cache\") pod \"premium-w1-1a81ae-kserve-679cdd99b8-mf842\" (UID: \"d40a938f-1aa7-4897-b5a7-91257132baf7\") " pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:38:01.874980 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.874929 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-tmp-dir\") pod \"premium-w1-1a81ae-kserve-679cdd99b8-mf842\" (UID: \"d40a938f-1aa7-4897-b5a7-91257132baf7\") " pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:38:01.876843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.876815 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-dshm\") pod \"premium-w1-1a81ae-kserve-679cdd99b8-mf842\" (UID: \"d40a938f-1aa7-4897-b5a7-91257132baf7\") " pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:38:01.877087 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.877070 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d40a938f-1aa7-4897-b5a7-91257132baf7-tls-certs\") pod \"premium-w1-1a81ae-kserve-679cdd99b8-mf842\" (UID: \"d40a938f-1aa7-4897-b5a7-91257132baf7\") " pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:38:01.886893 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.886872 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hrg6\" (UniqueName: \"kubernetes.io/projected/d40a938f-1aa7-4897-b5a7-91257132baf7-kube-api-access-6hrg6\") pod \"premium-w1-1a81ae-kserve-679cdd99b8-mf842\" (UID: \"d40a938f-1aa7-4897-b5a7-91257132baf7\") " pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:38:01.977735 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:01.977675 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:38:02.033988 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:02.033954 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:02.126869 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:02.126843 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:38:02.256413 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:02.256389 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842"] Oct 30 21:38:02.258127 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:38:02.258103 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd40a938f_1aa7_4897_b5a7_91257132baf7.slice/crio-b4b4d0ef1e25803e4c0b6f1fc7f76f0da2cdfa4818f537536d511fe5dc2f9109 WatchSource:0}: Error finding container b4b4d0ef1e25803e4c0b6f1fc7f76f0da2cdfa4818f537536d511fe5dc2f9109: Status 404 returned error can't find the container with id b4b4d0ef1e25803e4c0b6f1fc7f76f0da2cdfa4818f537536d511fe5dc2f9109 Oct 30 21:38:02.751270 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:02.751175 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:03.124146 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:03.124111 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" event={"ID":"d40a938f-1aa7-4897-b5a7-91257132baf7","Type":"ContainerStarted","Data":"290f91d4dfd4a1e83f4363e8306afeb90b1a4dd2916a01c86bfe4abc5f6dd870"} Oct 30 21:38:03.124298 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:03.124153 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" event={"ID":"d40a938f-1aa7-4897-b5a7-91257132baf7","Type":"ContainerStarted","Data":"b4b4d0ef1e25803e4c0b6f1fc7f76f0da2cdfa4818f537536d511fe5dc2f9109"} Oct 30 21:38:03.145471 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:03.145424 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podStartSLOduration=2.145410817 podStartE2EDuration="2.145410817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:03.143272475 +0000 UTC m=+551.739089152" watchObservedRunningTime="2026-10-30 21:38:03.145410817 +0000 UTC m=+551.741227477" Oct 30 21:38:03.654159 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:03.654126 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:03.978102 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:03.978026 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:38:04.543309 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:04.543275 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:05.355681 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:05.355643 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:06.200465 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:06.200429 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:06.919001 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:06.918972 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:07.635231 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:07.635200 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:08.242989 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:08.242950 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:08.261904 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:08.261872 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:38:08.274219 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:08.274193 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:38:08.453140 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:08.453101 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:09.158690 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:09.158651 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:09.171110 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:09.171089 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:38:09.932314 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:09.932275 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:10.654867 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:10.654832 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:11.648679 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:11.648640 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:11.977942 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:11.977855 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:38:12.000963 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:12.000935 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:38:12.189405 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:12.189367 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:38:12.551627 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:12.551592 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:13.269091 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:13.269018 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:13.876404 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:13.876372 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq"] Oct 30 21:38:13.880283 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:13.880259 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:38:13.888826 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:13.888803 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1d8034\"/\"premium-w3-1d8034-kserve-self-signed-certs\"" Oct 30 21:38:13.891643 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:13.891619 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq"] Oct 30 21:38:13.901513 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:13.901490 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-dshm\") pod \"premium-w3-1d8034-kserve-76c7f96f45-hsqbq\" (UID: \"ea440bed-5030-4eeb-85b7-1916ffe802af\") " pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:38:13.901605 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:13.901525 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-home\") pod \"premium-w3-1d8034-kserve-76c7f96f45-hsqbq\" (UID: \"ea440bed-5030-4eeb-85b7-1916ffe802af\") " pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:38:13.901785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:13.901746 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea440bed-5030-4eeb-85b7-1916ffe802af-tls-certs\") pod \"premium-w3-1d8034-kserve-76c7f96f45-hsqbq\" (UID: \"ea440bed-5030-4eeb-85b7-1916ffe802af\") " pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:38:13.901844 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:13.901826 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-tmp-dir\") pod \"premium-w3-1d8034-kserve-76c7f96f45-hsqbq\" (UID: \"ea440bed-5030-4eeb-85b7-1916ffe802af\") " pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:38:13.901880 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:13.901870 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-model-cache\") pod \"premium-w3-1d8034-kserve-76c7f96f45-hsqbq\" (UID: \"ea440bed-5030-4eeb-85b7-1916ffe802af\") " pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:38:13.901913 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:13.901899 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qr2j\" (UniqueName: \"kubernetes.io/projected/ea440bed-5030-4eeb-85b7-1916ffe802af-kube-api-access-2qr2j\") pod \"premium-w3-1d8034-kserve-76c7f96f45-hsqbq\" (UID: \"ea440bed-5030-4eeb-85b7-1916ffe802af\") " pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:38:13.924300 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:13.924269 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:14.003247 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:14.003215 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea440bed-5030-4eeb-85b7-1916ffe802af-tls-certs\") pod \"premium-w3-1d8034-kserve-76c7f96f45-hsqbq\" (UID: \"ea440bed-5030-4eeb-85b7-1916ffe802af\") " pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:38:14.003247 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:14.003249 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-tmp-dir\") pod \"premium-w3-1d8034-kserve-76c7f96f45-hsqbq\" (UID: \"ea440bed-5030-4eeb-85b7-1916ffe802af\") " pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:38:14.003447 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:14.003274 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-model-cache\") pod \"premium-w3-1d8034-kserve-76c7f96f45-hsqbq\" (UID: \"ea440bed-5030-4eeb-85b7-1916ffe802af\") " pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:38:14.003447 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:14.003300 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qr2j\" (UniqueName: \"kubernetes.io/projected/ea440bed-5030-4eeb-85b7-1916ffe802af-kube-api-access-2qr2j\") pod \"premium-w3-1d8034-kserve-76c7f96f45-hsqbq\" (UID: \"ea440bed-5030-4eeb-85b7-1916ffe802af\") " pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:38:14.003447 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:14.003323 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-dshm\") pod \"premium-w3-1d8034-kserve-76c7f96f45-hsqbq\" (UID: \"ea440bed-5030-4eeb-85b7-1916ffe802af\") " pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:38:14.003447 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:14.003342 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-home\") pod \"premium-w3-1d8034-kserve-76c7f96f45-hsqbq\" (UID: \"ea440bed-5030-4eeb-85b7-1916ffe802af\") " pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:38:14.003689 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:14.003666 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-tmp-dir\") pod \"premium-w3-1d8034-kserve-76c7f96f45-hsqbq\" (UID: \"ea440bed-5030-4eeb-85b7-1916ffe802af\") " pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:38:14.003741 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:14.003712 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-model-cache\") pod \"premium-w3-1d8034-kserve-76c7f96f45-hsqbq\" (UID: \"ea440bed-5030-4eeb-85b7-1916ffe802af\") " pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:38:14.003804 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:14.003783 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-home\") pod \"premium-w3-1d8034-kserve-76c7f96f45-hsqbq\" (UID: \"ea440bed-5030-4eeb-85b7-1916ffe802af\") " pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:38:14.005546 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:14.005522 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-dshm\") pod \"premium-w3-1d8034-kserve-76c7f96f45-hsqbq\" (UID: \"ea440bed-5030-4eeb-85b7-1916ffe802af\") " pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:38:14.005938 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:14.005919 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea440bed-5030-4eeb-85b7-1916ffe802af-tls-certs\") pod \"premium-w3-1d8034-kserve-76c7f96f45-hsqbq\" (UID: \"ea440bed-5030-4eeb-85b7-1916ffe802af\") " pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:38:14.019015 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:14.018992 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qr2j\" (UniqueName: \"kubernetes.io/projected/ea440bed-5030-4eeb-85b7-1916ffe802af-kube-api-access-2qr2j\") pod \"premium-w3-1d8034-kserve-76c7f96f45-hsqbq\" (UID: \"ea440bed-5030-4eeb-85b7-1916ffe802af\") " pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:38:14.026100 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:14.026079 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:14.193649 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:14.193565 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:38:14.473504 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:14.473473 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq"] Oct 30 21:38:14.473850 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:38:14.473708 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea440bed_5030_4eeb_85b7_1916ffe802af.slice/crio-55463fafe334facd8cde88f781fe86c84bab9e31e3389ab156b2fefe0e972d80 WatchSource:0}: Error finding container 55463fafe334facd8cde88f781fe86c84bab9e31e3389ab156b2fefe0e972d80: Status 404 returned error can't find the container with id 55463fafe334facd8cde88f781fe86c84bab9e31e3389ab156b2fefe0e972d80 Oct 30 21:38:14.659868 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:14.659834 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:15.196027 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:15.195989 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" event={"ID":"ea440bed-5030-4eeb-85b7-1916ffe802af","Type":"ContainerStarted","Data":"57ccf884c1ddf8e67ff6d99db1412fd71c1edce8a202df0175e22fcc2e964029"} Oct 30 21:38:15.196211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:15.196038 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" event={"ID":"ea440bed-5030-4eeb-85b7-1916ffe802af","Type":"ContainerStarted","Data":"55463fafe334facd8cde88f781fe86c84bab9e31e3389ab156b2fefe0e972d80"} Oct 30 21:38:15.218124 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:15.218071 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podStartSLOduration=2.21805784 podStartE2EDuration="2.21805784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:15.215874188 +0000 UTC m=+563.811690849" watchObservedRunningTime="2026-10-30 21:38:15.21805784 +0000 UTC m=+563.813874499" Oct 30 21:38:15.374620 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:15.374579 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:16.056531 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:16.056496 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:16.194281 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:16.194252 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:38:16.752629 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:16.752596 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:17.593059 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:17.593020 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:18.553326 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:18.553288 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:19.318783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:19.318727 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:20.339784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:20.339724 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:21.117328 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:21.117294 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:22.149440 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:22.149403 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:23.133922 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:23.133882 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:23.998104 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:23.998071 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:24.194470 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:24.194440 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:38:24.206687 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:24.206662 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:38:24.263111 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:24.263029 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:38:24.826304 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:24.826267 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:25.634970 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:25.634939 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:26.336016 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:26.335980 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:27.254674 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:27.254628 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:28.141525 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:28.141490 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:28.834813 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:28.834774 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:29.541220 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:29.541187 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:30.238731 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:30.238683 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:30.279375 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:38:30.279323 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:38:30.940911 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:30.940876 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:31.754060 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:31.754027 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:32.435092 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:32.435057 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:33.152501 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:33.152465 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:33.842702 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:33.842664 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:34.553157 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:34.553118 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:35.461038 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:35.460982 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:36.340964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:36.340930 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:37.253113 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:37.253071 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:38.146455 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:38.146420 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:38.713612 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:38.713581 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55412139-xqhwj"] Oct 30 21:38:38.717242 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:38.717226 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55412139-xqhwj" Oct 30 21:38:38.723705 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:38.723678 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:38:38.725442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:38.725418 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:38:38.725927 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:38.725907 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:38:38.728684 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:38.728664 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55412139-xqhwj"] Oct 30 21:38:38.780904 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:38.780879 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed020689-4336-4043-8e4a-dd496a48d080-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55412139-xqhwj\" (UID: \"ed020689-4336-4043-8e4a-dd496a48d080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55412139-xqhwj" Oct 30 21:38:38.781010 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:38.780908 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed020689-4336-4043-8e4a-dd496a48d080-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55412139-xqhwj\" (UID: \"ed020689-4336-4043-8e4a-dd496a48d080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55412139-xqhwj" Oct 30 21:38:38.781010 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:38.780940 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5pd7\" (UniqueName: \"kubernetes.io/projected/ed020689-4336-4043-8e4a-dd496a48d080-kube-api-access-m5pd7\") pod \"maas-api-revoke-sub-55412139-xqhwj\" (UID: \"ed020689-4336-4043-8e4a-dd496a48d080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55412139-xqhwj" Oct 30 21:38:38.824074 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:38.824055 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:38:38.882302 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:38.882276 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed020689-4336-4043-8e4a-dd496a48d080-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55412139-xqhwj\" (UID: \"ed020689-4336-4043-8e4a-dd496a48d080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55412139-xqhwj" Oct 30 21:38:38.882408 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:38.882307 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed020689-4336-4043-8e4a-dd496a48d080-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55412139-xqhwj\" (UID: \"ed020689-4336-4043-8e4a-dd496a48d080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55412139-xqhwj" Oct 30 21:38:38.882408 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:38.882340 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5pd7\" (UniqueName: \"kubernetes.io/projected/ed020689-4336-4043-8e4a-dd496a48d080-kube-api-access-m5pd7\") pod \"maas-api-revoke-sub-55412139-xqhwj\" (UID: \"ed020689-4336-4043-8e4a-dd496a48d080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55412139-xqhwj" Oct 30 21:38:38.882917 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:38.882899 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed020689-4336-4043-8e4a-dd496a48d080-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55412139-xqhwj\" (UID: \"ed020689-4336-4043-8e4a-dd496a48d080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55412139-xqhwj" Oct 30 21:38:38.884631 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:38.884611 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed020689-4336-4043-8e4a-dd496a48d080-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55412139-xqhwj\" (UID: \"ed020689-4336-4043-8e4a-dd496a48d080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55412139-xqhwj" Oct 30 21:38:38.901450 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:38.901418 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5pd7\" (UniqueName: \"kubernetes.io/projected/ed020689-4336-4043-8e4a-dd496a48d080-kube-api-access-m5pd7\") pod \"maas-api-revoke-sub-55412139-xqhwj\" (UID: \"ed020689-4336-4043-8e4a-dd496a48d080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55412139-xqhwj" Oct 30 21:38:39.031705 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:39.031624 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55412139-xqhwj" Oct 30 21:38:39.135976 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:39.135947 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:39.309071 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:39.309043 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55412139-xqhwj"] Oct 30 21:38:39.311171 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:38:39.311145 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded020689_4336_4043_8e4a_dd496a48d080.slice/crio-69dd4c85799c24214bf465058b54ebdad78324fc8f4578aafa7adcf0875eafb5 WatchSource:0}: Error finding container 69dd4c85799c24214bf465058b54ebdad78324fc8f4578aafa7adcf0875eafb5: Status 404 returned error can't find the container with id 69dd4c85799c24214bf465058b54ebdad78324fc8f4578aafa7adcf0875eafb5 Oct 30 21:38:39.313266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:39.313247 5801 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:38:39.362883 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:39.362848 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55412139-xqhwj" event={"ID":"ed020689-4336-4043-8e4a-dd496a48d080","Type":"ContainerStarted","Data":"69dd4c85799c24214bf465058b54ebdad78324fc8f4578aafa7adcf0875eafb5"} Oct 30 21:38:40.054182 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:40.054151 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:40.723925 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:40.723847 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:40.959345 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:40.959300 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:41.651180 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:41.651150 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:42.338828 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:42.338798 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:42.382393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:42.382361 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55412139-xqhwj" event={"ID":"ed020689-4336-4043-8e4a-dd496a48d080","Type":"ContainerStarted","Data":"62597f6ec99dd8bface1e458a080a36620ceb87dfce0645ddf90987701cef2c1"} Oct 30 21:38:42.402602 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:42.402534 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55412139-xqhwj" podStartSLOduration=1.763913346 podStartE2EDuration="4.402515438s" totalImagesPullingTime="2.638602092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:42.401526455 +0000 UTC m=+590.997343132" watchObservedRunningTime="2026-10-30 21:38:42.402515438 +0000 UTC m=+590.998332099" Oct 30 21:38:43.295880 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:43.295795 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:43.389864 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:43.389833 5801 generic.go:356] "Generic (PLEG): container finished" podID="ed020689-4336-4043-8e4a-dd496a48d080" containerID="62597f6ec99dd8bface1e458a080a36620ceb87dfce0645ddf90987701cef2c1" exitCode=0 Oct 30 21:38:43.390232 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:43.389918 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55412139-xqhwj" event={"ID":"ed020689-4336-4043-8e4a-dd496a48d080","Type":"ContainerDied","Data":"62597f6ec99dd8bface1e458a080a36620ceb87dfce0645ddf90987701cef2c1"} Oct 30 21:38:44.137017 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:44.136989 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:44.536258 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:44.536232 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55412139-xqhwj" Oct 30 21:38:44.650371 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:44.650288 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed020689-4336-4043-8e4a-dd496a48d080-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed020689-4336-4043-8e4a-dd496a48d080-maas-api-xks-ca\") pod \"ed020689-4336-4043-8e4a-dd496a48d080\" (UID: \"ed020689-4336-4043-8e4a-dd496a48d080\") " Oct 30 21:38:44.650371 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:44.650335 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed020689-4336-4043-8e4a-dd496a48d080-kube-api-access-m5pd7\" (UniqueName: \"kubernetes.io/projected/ed020689-4336-4043-8e4a-dd496a48d080-kube-api-access-m5pd7\") pod \"ed020689-4336-4043-8e4a-dd496a48d080\" (UID: \"ed020689-4336-4043-8e4a-dd496a48d080\") " Oct 30 21:38:44.650616 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:44.650424 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed020689-4336-4043-8e4a-dd496a48d080-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed020689-4336-4043-8e4a-dd496a48d080-maas-api-openshift-ca\") pod \"ed020689-4336-4043-8e4a-dd496a48d080\" (UID: \"ed020689-4336-4043-8e4a-dd496a48d080\") " Oct 30 21:38:44.650838 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:44.650803 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed020689-4336-4043-8e4a-dd496a48d080-maas-api-openshift-ca" pod "ed020689-4336-4043-8e4a-dd496a48d080" (UID: "ed020689-4336-4043-8e4a-dd496a48d080"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:44.652601 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:44.652578 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed020689-4336-4043-8e4a-dd496a48d080-maas-api-xks-ca" pod "ed020689-4336-4043-8e4a-dd496a48d080" (UID: "ed020689-4336-4043-8e4a-dd496a48d080"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:44.652601 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:44.652586 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed020689-4336-4043-8e4a-dd496a48d080-kube-api-access-m5pd7" pod "ed020689-4336-4043-8e4a-dd496a48d080" (UID: "ed020689-4336-4043-8e4a-dd496a48d080"). InnerVolumeSpecName "kube-api-access-m5pd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:44.751986 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:44.751935 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed020689-4336-4043-8e4a-dd496a48d080-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:38:44.751986 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:44.751982 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed020689-4336-4043-8e4a-dd496a48d080-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:38:44.751986 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:44.751997 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5pd7\" (UniqueName: \"kubernetes.io/projected/ed020689-4336-4043-8e4a-dd496a48d080-kube-api-access-m5pd7\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:38:44.852937 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:44.852897 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:45.402157 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:45.402126 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55412139-xqhwj" Oct 30 21:38:45.402336 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:45.402157 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55412139-xqhwj" event={"ID":"ed020689-4336-4043-8e4a-dd496a48d080","Type":"ContainerDied","Data":"69dd4c85799c24214bf465058b54ebdad78324fc8f4578aafa7adcf0875eafb5"} Oct 30 21:38:45.402336 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:45.402210 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69dd4c85799c24214bf465058b54ebdad78324fc8f4578aafa7adcf0875eafb5" Oct 30 21:38:45.548950 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:45.548906 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:46.250664 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:46.250601 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:46.931720 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:46.931684 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:47.738618 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:47.738557 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:48.535362 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:48.535326 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:49.268635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:49.268598 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:49.820745 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:49.820715 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-515c34ba-crmc2"] Oct 30 21:38:49.821207 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:49.821191 5801 cpu_manager.go:414] "containerMap: removing container" podUID="ed020689-4336-4043-8e4a-dd496a48d080" containerName="revoke-keys" Oct 30 21:38:49.821250 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:49.821210 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed020689-4336-4043-8e4a-dd496a48d080" containerName="revoke-keys" Oct 30 21:38:49.821300 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:49.821292 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed020689-4336-4043-8e4a-dd496a48d080" containerName="revoke-keys" Oct 30 21:38:49.824592 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:49.824577 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c34ba-crmc2" Oct 30 21:38:49.831877 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:49.831851 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:38:49.832171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:49.832150 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:38:49.832347 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:49.832315 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:38:49.832444 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:49.831779 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:38:49.834905 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:49.834884 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-515c34ba-crmc2"] Oct 30 21:38:49.908375 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:49.908341 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc2deb4c-ba08-4812-9c4b-886b5c7a192d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-515c34ba-crmc2\" (UID: \"fc2deb4c-ba08-4812-9c4b-886b5c7a192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c34ba-crmc2" Oct 30 21:38:49.908560 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:49.908408 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72x65\" (UniqueName: \"kubernetes.io/projected/fc2deb4c-ba08-4812-9c4b-886b5c7a192d-kube-api-access-72x65\") pod \"maas-api-revoke-sub-515c34ba-crmc2\" (UID: \"fc2deb4c-ba08-4812-9c4b-886b5c7a192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c34ba-crmc2" Oct 30 21:38:49.908615 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:49.908550 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc2deb4c-ba08-4812-9c4b-886b5c7a192d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-515c34ba-crmc2\" (UID: \"fc2deb4c-ba08-4812-9c4b-886b5c7a192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c34ba-crmc2" Oct 30 21:38:50.009308 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:50.009278 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc2deb4c-ba08-4812-9c4b-886b5c7a192d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-515c34ba-crmc2\" (UID: \"fc2deb4c-ba08-4812-9c4b-886b5c7a192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c34ba-crmc2" Oct 30 21:38:50.009480 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:50.009346 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72x65\" (UniqueName: \"kubernetes.io/projected/fc2deb4c-ba08-4812-9c4b-886b5c7a192d-kube-api-access-72x65\") pod \"maas-api-revoke-sub-515c34ba-crmc2\" (UID: \"fc2deb4c-ba08-4812-9c4b-886b5c7a192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c34ba-crmc2" Oct 30 21:38:50.009480 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:50.009399 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc2deb4c-ba08-4812-9c4b-886b5c7a192d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-515c34ba-crmc2\" (UID: \"fc2deb4c-ba08-4812-9c4b-886b5c7a192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c34ba-crmc2" Oct 30 21:38:50.009944 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:50.009921 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc2deb4c-ba08-4812-9c4b-886b5c7a192d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-515c34ba-crmc2\" (UID: \"fc2deb4c-ba08-4812-9c4b-886b5c7a192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c34ba-crmc2" Oct 30 21:38:50.011710 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:50.011686 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc2deb4c-ba08-4812-9c4b-886b5c7a192d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-515c34ba-crmc2\" (UID: \"fc2deb4c-ba08-4812-9c4b-886b5c7a192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c34ba-crmc2" Oct 30 21:38:50.021666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:50.021645 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72x65\" (UniqueName: \"kubernetes.io/projected/fc2deb4c-ba08-4812-9c4b-886b5c7a192d-kube-api-access-72x65\") pod \"maas-api-revoke-sub-515c34ba-crmc2\" (UID: \"fc2deb4c-ba08-4812-9c4b-886b5c7a192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c34ba-crmc2" Oct 30 21:38:50.030398 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:50.030372 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:50.140019 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:50.139931 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c34ba-crmc2" Oct 30 21:38:50.417107 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:50.417080 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-515c34ba-crmc2"] Oct 30 21:38:50.419171 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:38:50.419145 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2deb4c_ba08_4812_9c4b_886b5c7a192d.slice/crio-31e9c02e226d595350ff1bacefa1585db945b2c690db804f9f846f7a69326482 WatchSource:0}: Error finding container 31e9c02e226d595350ff1bacefa1585db945b2c690db804f9f846f7a69326482: Status 404 returned error can't find the container with id 31e9c02e226d595350ff1bacefa1585db945b2c690db804f9f846f7a69326482 Oct 30 21:38:50.431175 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:50.431142 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c34ba-crmc2" event={"ID":"fc2deb4c-ba08-4812-9c4b-886b5c7a192d","Type":"ContainerStarted","Data":"31e9c02e226d595350ff1bacefa1585db945b2c690db804f9f846f7a69326482"} Oct 30 21:38:50.751348 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:50.751270 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:51.438462 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:51.438372 5801 generic.go:356] "Generic (PLEG): container finished" podID="fc2deb4c-ba08-4812-9c4b-886b5c7a192d" containerID="90378d517d91838283de2b675dc87a9d89efd80603734700fae672061ee1a964" exitCode=0 Oct 30 21:38:51.438870 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:51.438466 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c34ba-crmc2" event={"ID":"fc2deb4c-ba08-4812-9c4b-886b5c7a192d","Type":"ContainerDied","Data":"90378d517d91838283de2b675dc87a9d89efd80603734700fae672061ee1a964"} Oct 30 21:38:51.447219 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:51.447193 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:52.142671 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:52.142637 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:52.593167 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:52.593142 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c34ba-crmc2" Oct 30 21:38:52.741260 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:52.741178 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc2deb4c-ba08-4812-9c4b-886b5c7a192d-kube-api-access-72x65\" (UniqueName: \"kubernetes.io/projected/fc2deb4c-ba08-4812-9c4b-886b5c7a192d-kube-api-access-72x65\") pod \"fc2deb4c-ba08-4812-9c4b-886b5c7a192d\" (UID: \"fc2deb4c-ba08-4812-9c4b-886b5c7a192d\") " Oct 30 21:38:52.741260 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:52.741242 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc2deb4c-ba08-4812-9c4b-886b5c7a192d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc2deb4c-ba08-4812-9c4b-886b5c7a192d-maas-api-xks-ca\") pod \"fc2deb4c-ba08-4812-9c4b-886b5c7a192d\" (UID: \"fc2deb4c-ba08-4812-9c4b-886b5c7a192d\") " Oct 30 21:38:52.741456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:52.741393 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc2deb4c-ba08-4812-9c4b-886b5c7a192d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc2deb4c-ba08-4812-9c4b-886b5c7a192d-maas-api-openshift-ca\") pod \"fc2deb4c-ba08-4812-9c4b-886b5c7a192d\" (UID: \"fc2deb4c-ba08-4812-9c4b-886b5c7a192d\") " Oct 30 21:38:52.741734 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:52.741707 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc2deb4c-ba08-4812-9c4b-886b5c7a192d-maas-api-openshift-ca" pod "fc2deb4c-ba08-4812-9c4b-886b5c7a192d" (UID: "fc2deb4c-ba08-4812-9c4b-886b5c7a192d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:52.743307 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:52.743283 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc2deb4c-ba08-4812-9c4b-886b5c7a192d-kube-api-access-72x65" pod "fc2deb4c-ba08-4812-9c4b-886b5c7a192d" (UID: "fc2deb4c-ba08-4812-9c4b-886b5c7a192d"). InnerVolumeSpecName "kube-api-access-72x65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:52.743307 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:52.743294 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc2deb4c-ba08-4812-9c4b-886b5c7a192d-maas-api-xks-ca" pod "fc2deb4c-ba08-4812-9c4b-886b5c7a192d" (UID: "fc2deb4c-ba08-4812-9c4b-886b5c7a192d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:52.837448 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:52.837414 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:52.843079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:52.843054 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc2deb4c-ba08-4812-9c4b-886b5c7a192d-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:38:52.843177 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:52.843082 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc2deb4c-ba08-4812-9c4b-886b5c7a192d-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:38:52.843177 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:52.843093 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72x65\" (UniqueName: \"kubernetes.io/projected/fc2deb4c-ba08-4812-9c4b-886b5c7a192d-kube-api-access-72x65\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:38:53.451519 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.451492 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c34ba-crmc2" Oct 30 21:38:53.451709 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.451526 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-515c34ba-crmc2" event={"ID":"fc2deb4c-ba08-4812-9c4b-886b5c7a192d","Type":"ContainerDied","Data":"31e9c02e226d595350ff1bacefa1585db945b2c690db804f9f846f7a69326482"} Oct 30 21:38:53.451709 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.451597 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31e9c02e226d595350ff1bacefa1585db945b2c690db804f9f846f7a69326482" Oct 30 21:38:53.490577 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.490538 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5"] Oct 30 21:38:53.491079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.491062 5801 cpu_manager.go:414] "containerMap: removing container" podUID="fc2deb4c-ba08-4812-9c4b-886b5c7a192d" containerName="revoke-keys" Oct 30 21:38:53.491079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.491080 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc2deb4c-ba08-4812-9c4b-886b5c7a192d" containerName="revoke-keys" Oct 30 21:38:53.491195 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.491153 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc2deb4c-ba08-4812-9c4b-886b5c7a192d" containerName="revoke-keys" Oct 30 21:38:53.494439 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.494420 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.498070 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.498048 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-1d8034-openshift-default-dockercfg-hl29t\"" Oct 30 21:38:53.508946 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.507081 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5"] Oct 30 21:38:53.649176 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.649131 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:53.650780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.650742 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-data\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.650888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.650863 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-envoy\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.650931 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.650904 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-workload-certs\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.650975 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.650962 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-credential-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.651057 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.651045 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-token\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.651195 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.651178 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-workload-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.651254 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.651205 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istiod-ca-cert\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.651254 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.651223 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msv4z\" (UniqueName: \"kubernetes.io/projected/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-kube-api-access-msv4z\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.651568 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.651550 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-podinfo\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.737098 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.737021 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr"] Oct 30 21:38:53.740746 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.740730 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.748373 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.748345 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-db0b54-openshift-default-dockercfg-ltx2s\"" Oct 30 21:38:53.752628 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.752609 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-token\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.752717 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.752668 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-workload-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.752717 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.752693 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istiod-ca-cert\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.752717 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.752708 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msv4z\" (UniqueName: \"kubernetes.io/projected/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-kube-api-access-msv4z\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.752867 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.752731 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-podinfo\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.752867 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.752778 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-data\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.752867 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.752852 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-envoy\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.753017 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.752895 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-workload-certs\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.753017 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.752937 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-credential-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.753151 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.753129 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-workload-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.753302 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.753267 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-credential-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.753302 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.753294 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-data\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.753458 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.753437 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-workload-certs\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.753511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.753496 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istiod-ca-cert\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.755142 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.755124 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-envoy\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.755288 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.755271 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-podinfo\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.778010 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.777989 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-token\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.778138 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.778120 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msv4z\" (UniqueName: \"kubernetes.io/projected/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-kube-api-access-msv4z\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.796129 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.796106 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr"] Oct 30 21:38:53.807934 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.807912 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:53.854295 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.854265 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-istio-data\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.854444 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.854305 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgwc4\" (UniqueName: \"kubernetes.io/projected/24b939ed-32e5-407e-a919-c0395d914dcd-kube-api-access-fgwc4\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.854444 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.854421 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b939ed-32e5-407e-a919-c0395d914dcd-istiod-ca-cert\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.854532 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.854513 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b939ed-32e5-407e-a919-c0395d914dcd-istio-token\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.854589 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.854574 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-workload-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.854649 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.854629 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-istio-envoy\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.854766 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.854656 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-workload-certs\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.854766 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.854716 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b939ed-32e5-407e-a919-c0395d914dcd-istio-podinfo\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.854897 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.854805 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-credential-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.955779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.955711 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b939ed-32e5-407e-a919-c0395d914dcd-istiod-ca-cert\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.955954 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.955816 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b939ed-32e5-407e-a919-c0395d914dcd-istio-token\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.955954 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.955852 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-workload-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.955954 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.955874 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-istio-envoy\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.955954 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.955892 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-workload-certs\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.955954 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.955941 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b939ed-32e5-407e-a919-c0395d914dcd-istio-podinfo\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.956285 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.956264 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-workload-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.956349 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.956331 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-workload-certs\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.956444 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.956427 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-credential-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.956529 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.956484 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-istio-data\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.956582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.956524 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgwc4\" (UniqueName: \"kubernetes.io/projected/24b939ed-32e5-407e-a919-c0395d914dcd-kube-api-access-fgwc4\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.956582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.956529 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b939ed-32e5-407e-a919-c0395d914dcd-istiod-ca-cert\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.956892 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.956874 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-istio-data\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.956994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.956857 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-credential-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.958393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.958364 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-istio-envoy\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.958541 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.958522 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b939ed-32e5-407e-a919-c0395d914dcd-istio-podinfo\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.969172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.969153 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b939ed-32e5-407e-a919-c0395d914dcd-istio-token\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:53.969309 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:53.969292 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgwc4\" (UniqueName: \"kubernetes.io/projected/24b939ed-32e5-407e-a919-c0395d914dcd-kube-api-access-fgwc4\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:54.053343 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:54.053313 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:54.101023 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:54.100992 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5"] Oct 30 21:38:54.103028 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:38:54.102998 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fcdad2f_d069_41a7_ace4_49d0ba4abfaa.slice/crio-87e82958e8ed8dbe3834959ac713d3519ef243222b443fd9be267be2cd1b6175 WatchSource:0}: Error finding container 87e82958e8ed8dbe3834959ac713d3519ef243222b443fd9be267be2cd1b6175: Status 404 returned error can't find the container with id 87e82958e8ed8dbe3834959ac713d3519ef243222b443fd9be267be2cd1b6175 Oct 30 21:38:54.106257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:54.106218 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:38:54.106362 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:54.106306 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:38:54.106362 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:54.106351 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:38:54.338644 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:54.338603 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr"] Oct 30 21:38:54.340991 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:38:54.340954 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b939ed_32e5_407e_a919_c0395d914dcd.slice/crio-b066fe451c230f8d3bf2ca3ea17da60c1ba926ac67cd720295e56f26d9cdbd08 WatchSource:0}: Error finding container b066fe451c230f8d3bf2ca3ea17da60c1ba926ac67cd720295e56f26d9cdbd08: Status 404 returned error can't find the container with id b066fe451c230f8d3bf2ca3ea17da60c1ba926ac67cd720295e56f26d9cdbd08 Oct 30 21:38:54.344125 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:54.344087 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:38:54.344208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:54.344169 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:38:54.344253 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:54.344207 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:38:54.460238 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:54.460199 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" event={"ID":"24b939ed-32e5-407e-a919-c0395d914dcd","Type":"ContainerStarted","Data":"e63e393f18cc121add112014195a2b67d7f6c874d342e7b86314fe7bbbdf0237"} Oct 30 21:38:54.460462 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:54.460264 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" event={"ID":"24b939ed-32e5-407e-a919-c0395d914dcd","Type":"ContainerStarted","Data":"b066fe451c230f8d3bf2ca3ea17da60c1ba926ac67cd720295e56f26d9cdbd08"} Oct 30 21:38:54.462365 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:54.462338 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" event={"ID":"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa","Type":"ContainerStarted","Data":"6bbadfdda2e87646a87cad137968f08f6b0e9b7972e31f1fc58d65c2eb5f43f0"} Oct 30 21:38:54.462576 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:54.462545 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" event={"ID":"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa","Type":"ContainerStarted","Data":"87e82958e8ed8dbe3834959ac713d3519ef243222b443fd9be267be2cd1b6175"} Oct 30 21:38:54.490256 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:54.490193 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" podStartSLOduration=1.4901749930000001 podStartE2EDuration="1.490174993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:54.489209305 +0000 UTC m=+603.085025965" watchObservedRunningTime="2026-10-30 21:38:54.490174993 +0000 UTC m=+603.085991670" Oct 30 21:38:54.512859 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:54.512585 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" podStartSLOduration=1.512566793 podStartE2EDuration="1.512566793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:54.512088021 +0000 UTC m=+603.107904718" watchObservedRunningTime="2026-10-30 21:38:54.512566793 +0000 UTC m=+603.108383457" Oct 30 21:38:54.641057 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:54.640961 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:54.808472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:54.808441 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:38:55.062001 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.057121 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:38:55.062933 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.062896 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:38:55.063198 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.063171 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" podUID="24b939ed-32e5-407e-a919-c0395d914dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:38:55.310617 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.310577 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e04ca9a6-p8zg4"] Oct 30 21:38:55.315032 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.314861 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04ca9a6-p8zg4" Oct 30 21:38:55.318507 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.318465 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e04ca9a6-p8zg4"] Oct 30 21:38:55.319465 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.319448 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:38:55.319935 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.319918 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:38:55.319999 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.319982 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:38:55.320096 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.320076 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:38:55.509447 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.509401 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c751727a-5224-4b96-9a94-2430a78bd46a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e04ca9a6-p8zg4\" (UID: \"c751727a-5224-4b96-9a94-2430a78bd46a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04ca9a6-p8zg4" Oct 30 21:38:55.509661 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.509467 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzwfh\" (UniqueName: \"kubernetes.io/projected/c751727a-5224-4b96-9a94-2430a78bd46a-kube-api-access-fzwfh\") pod \"maas-api-revoke-sub-e04ca9a6-p8zg4\" (UID: \"c751727a-5224-4b96-9a94-2430a78bd46a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04ca9a6-p8zg4" Oct 30 21:38:55.509661 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.509539 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c751727a-5224-4b96-9a94-2430a78bd46a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e04ca9a6-p8zg4\" (UID: \"c751727a-5224-4b96-9a94-2430a78bd46a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04ca9a6-p8zg4" Oct 30 21:38:55.549415 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.549377 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:55.616840 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.616104 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c751727a-5224-4b96-9a94-2430a78bd46a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e04ca9a6-p8zg4\" (UID: \"c751727a-5224-4b96-9a94-2430a78bd46a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04ca9a6-p8zg4" Oct 30 21:38:55.616840 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.616372 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c751727a-5224-4b96-9a94-2430a78bd46a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e04ca9a6-p8zg4\" (UID: \"c751727a-5224-4b96-9a94-2430a78bd46a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04ca9a6-p8zg4" Oct 30 21:38:55.616840 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.616455 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzwfh\" (UniqueName: \"kubernetes.io/projected/c751727a-5224-4b96-9a94-2430a78bd46a-kube-api-access-fzwfh\") pod \"maas-api-revoke-sub-e04ca9a6-p8zg4\" (UID: \"c751727a-5224-4b96-9a94-2430a78bd46a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04ca9a6-p8zg4" Oct 30 21:38:55.620781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.618351 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c751727a-5224-4b96-9a94-2430a78bd46a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e04ca9a6-p8zg4\" (UID: \"c751727a-5224-4b96-9a94-2430a78bd46a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04ca9a6-p8zg4" Oct 30 21:38:55.623484 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.623420 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c751727a-5224-4b96-9a94-2430a78bd46a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e04ca9a6-p8zg4\" (UID: \"c751727a-5224-4b96-9a94-2430a78bd46a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04ca9a6-p8zg4" Oct 30 21:38:55.633961 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.633934 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzwfh\" (UniqueName: \"kubernetes.io/projected/c751727a-5224-4b96-9a94-2430a78bd46a-kube-api-access-fzwfh\") pod \"maas-api-revoke-sub-e04ca9a6-p8zg4\" (UID: \"c751727a-5224-4b96-9a94-2430a78bd46a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04ca9a6-p8zg4" Oct 30 21:38:55.810127 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.810072 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:38:55.810520 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.810146 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" podUID="1fcdad2f-d069-41a7-ace4-49d0ba4abfaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 21:38:55.848926 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.848891 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e644c29-jb62q"] Oct 30 21:38:55.884835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.865080 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e644c29-jb62q" Oct 30 21:38:55.923069 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.917969 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e644c29-jb62q"] Oct 30 21:38:55.923069 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.920870 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d5db44-d8e7-4c47-b9b0-1bc2d52c891b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e644c29-jb62q\" (UID: \"56d5db44-d8e7-4c47-b9b0-1bc2d52c891b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e644c29-jb62q" Oct 30 21:38:55.923069 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.920908 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d5db44-d8e7-4c47-b9b0-1bc2d52c891b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e644c29-jb62q\" (UID: \"56d5db44-d8e7-4c47-b9b0-1bc2d52c891b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e644c29-jb62q" Oct 30 21:38:55.923069 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.921033 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr77k\" (UniqueName: \"kubernetes.io/projected/56d5db44-d8e7-4c47-b9b0-1bc2d52c891b-kube-api-access-wr77k\") pod \"maas-api-revoke-sub-6e644c29-jb62q\" (UID: \"56d5db44-d8e7-4c47-b9b0-1bc2d52c891b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e644c29-jb62q" Oct 30 21:38:55.931570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:55.931540 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04ca9a6-p8zg4" Oct 30 21:38:56.022730 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.022699 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d5db44-d8e7-4c47-b9b0-1bc2d52c891b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e644c29-jb62q\" (UID: \"56d5db44-d8e7-4c47-b9b0-1bc2d52c891b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e644c29-jb62q" Oct 30 21:38:56.022730 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.022732 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d5db44-d8e7-4c47-b9b0-1bc2d52c891b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e644c29-jb62q\" (UID: \"56d5db44-d8e7-4c47-b9b0-1bc2d52c891b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e644c29-jb62q" Oct 30 21:38:56.022963 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.022827 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr77k\" (UniqueName: \"kubernetes.io/projected/56d5db44-d8e7-4c47-b9b0-1bc2d52c891b-kube-api-access-wr77k\") pod \"maas-api-revoke-sub-6e644c29-jb62q\" (UID: \"56d5db44-d8e7-4c47-b9b0-1bc2d52c891b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e644c29-jb62q" Oct 30 21:38:56.023337 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.023314 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d5db44-d8e7-4c47-b9b0-1bc2d52c891b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e644c29-jb62q\" (UID: \"56d5db44-d8e7-4c47-b9b0-1bc2d52c891b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e644c29-jb62q" Oct 30 21:38:56.025394 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.025368 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d5db44-d8e7-4c47-b9b0-1bc2d52c891b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e644c29-jb62q\" (UID: \"56d5db44-d8e7-4c47-b9b0-1bc2d52c891b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e644c29-jb62q" Oct 30 21:38:56.036672 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.036648 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr77k\" (UniqueName: \"kubernetes.io/projected/56d5db44-d8e7-4c47-b9b0-1bc2d52c891b-kube-api-access-wr77k\") pod \"maas-api-revoke-sub-6e644c29-jb62q\" (UID: \"56d5db44-d8e7-4c47-b9b0-1bc2d52c891b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e644c29-jb62q" Oct 30 21:38:56.084990 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.082190 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:38:56.084990 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.082246 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" podUID="24b939ed-32e5-407e-a919-c0395d914dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:38:56.238238 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.238158 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e644c29-jb62q" Oct 30 21:38:56.315354 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.315324 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e04ca9a6-p8zg4"] Oct 30 21:38:56.315427 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:38:56.315349 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc751727a_5224_4b96_9a94_2430a78bd46a.slice/crio-c1a92f299a11f3279147349f071850ab6ba6bc77adda7335519807b0e9bf3498 WatchSource:0}: Error finding container c1a92f299a11f3279147349f071850ab6ba6bc77adda7335519807b0e9bf3498: Status 404 returned error can't find the container with id c1a92f299a11f3279147349f071850ab6ba6bc77adda7335519807b0e9bf3498 Oct 30 21:38:56.337693 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.337666 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:56.512828 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.512551 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c3de97-w4wjs"] Oct 30 21:38:56.520729 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.520696 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3de97-w4wjs" Oct 30 21:38:56.528119 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.528090 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c3de97-w4wjs"] Oct 30 21:38:56.537772 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.537719 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04ca9a6-p8zg4" event={"ID":"c751727a-5224-4b96-9a94-2430a78bd46a","Type":"ContainerStarted","Data":"c1a92f299a11f3279147349f071850ab6ba6bc77adda7335519807b0e9bf3498"} Oct 30 21:38:56.612794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.612529 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e644c29-jb62q"] Oct 30 21:38:56.635272 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.635245 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e1bc47-ed65-41be-acf1-bfb2b33ab26f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80c3de97-w4wjs\" (UID: \"16e1bc47-ed65-41be-acf1-bfb2b33ab26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3de97-w4wjs" Oct 30 21:38:56.635426 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.635403 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8rqf\" (UniqueName: \"kubernetes.io/projected/16e1bc47-ed65-41be-acf1-bfb2b33ab26f-kube-api-access-p8rqf\") pod \"maas-api-revoke-sub-80c3de97-w4wjs\" (UID: \"16e1bc47-ed65-41be-acf1-bfb2b33ab26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3de97-w4wjs" Oct 30 21:38:56.636131 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.636106 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e1bc47-ed65-41be-acf1-bfb2b33ab26f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80c3de97-w4wjs\" (UID: \"16e1bc47-ed65-41be-acf1-bfb2b33ab26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3de97-w4wjs" Oct 30 21:38:56.736932 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.736891 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e1bc47-ed65-41be-acf1-bfb2b33ab26f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80c3de97-w4wjs\" (UID: \"16e1bc47-ed65-41be-acf1-bfb2b33ab26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3de97-w4wjs" Oct 30 21:38:56.737065 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.736988 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e1bc47-ed65-41be-acf1-bfb2b33ab26f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80c3de97-w4wjs\" (UID: \"16e1bc47-ed65-41be-acf1-bfb2b33ab26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3de97-w4wjs" Oct 30 21:38:56.737105 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.737082 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8rqf\" (UniqueName: \"kubernetes.io/projected/16e1bc47-ed65-41be-acf1-bfb2b33ab26f-kube-api-access-p8rqf\") pod \"maas-api-revoke-sub-80c3de97-w4wjs\" (UID: \"16e1bc47-ed65-41be-acf1-bfb2b33ab26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3de97-w4wjs" Oct 30 21:38:56.737550 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.737530 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e1bc47-ed65-41be-acf1-bfb2b33ab26f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80c3de97-w4wjs\" (UID: \"16e1bc47-ed65-41be-acf1-bfb2b33ab26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3de97-w4wjs" Oct 30 21:38:56.772837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.771771 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8rqf\" (UniqueName: \"kubernetes.io/projected/16e1bc47-ed65-41be-acf1-bfb2b33ab26f-kube-api-access-p8rqf\") pod \"maas-api-revoke-sub-80c3de97-w4wjs\" (UID: \"16e1bc47-ed65-41be-acf1-bfb2b33ab26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3de97-w4wjs" Oct 30 21:38:56.772837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.772119 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e1bc47-ed65-41be-acf1-bfb2b33ab26f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80c3de97-w4wjs\" (UID: \"16e1bc47-ed65-41be-acf1-bfb2b33ab26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3de97-w4wjs" Oct 30 21:38:56.809248 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.809206 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:38:56.809440 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.809274 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" podUID="1fcdad2f-d069-41a7-ace4-49d0ba4abfaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 21:38:56.847736 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.847706 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3de97-w4wjs" Oct 30 21:38:56.944728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.944693 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea80e817-pg2sx"] Oct 30 21:38:56.949847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:56.949824 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea80e817-pg2sx" Oct 30 21:38:57.019828 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.019792 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea80e817-pg2sx"] Oct 30 21:38:57.041403 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.040624 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea80e817-pg2sx\" (UID: \"9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea80e817-pg2sx" Oct 30 21:38:57.041403 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.040696 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea80e817-pg2sx\" (UID: \"9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea80e817-pg2sx" Oct 30 21:38:57.041403 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.040721 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67t26\" (UniqueName: \"kubernetes.io/projected/9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e-kube-api-access-67t26\") pod \"maas-api-revoke-sub-ea80e817-pg2sx\" (UID: \"9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea80e817-pg2sx" Oct 30 21:38:57.073576 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.065632 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:57.074806 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.074358 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:38:57.074806 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.074412 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" podUID="24b939ed-32e5-407e-a919-c0395d914dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:38:57.141886 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.141740 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67t26\" (UniqueName: \"kubernetes.io/projected/9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e-kube-api-access-67t26\") pod \"maas-api-revoke-sub-ea80e817-pg2sx\" (UID: \"9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea80e817-pg2sx" Oct 30 21:38:57.141886 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.141876 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea80e817-pg2sx\" (UID: \"9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea80e817-pg2sx" Oct 30 21:38:57.142046 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.141954 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea80e817-pg2sx\" (UID: \"9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea80e817-pg2sx" Oct 30 21:38:57.142538 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.142520 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea80e817-pg2sx\" (UID: \"9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea80e817-pg2sx" Oct 30 21:38:57.144081 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.144065 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea80e817-pg2sx\" (UID: \"9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea80e817-pg2sx" Oct 30 21:38:57.210640 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.210600 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67t26\" (UniqueName: \"kubernetes.io/projected/9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e-kube-api-access-67t26\") pod \"maas-api-revoke-sub-ea80e817-pg2sx\" (UID: \"9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea80e817-pg2sx" Oct 30 21:38:57.222456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.222426 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c3de97-w4wjs"] Oct 30 21:38:57.225548 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:38:57.225519 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e1bc47_ed65_41be_acf1_bfb2b33ab26f.slice/crio-d64846f8c076d4407cad00c467df8fe7113aec79f61d606b58b544245ecd6869 WatchSource:0}: Error finding container d64846f8c076d4407cad00c467df8fe7113aec79f61d606b58b544245ecd6869: Status 404 returned error can't find the container with id d64846f8c076d4407cad00c467df8fe7113aec79f61d606b58b544245ecd6869 Oct 30 21:38:57.230593 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.230566 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:57.342002 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.341967 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea80e817-pg2sx" Oct 30 21:38:57.549261 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.549231 5801 generic.go:356] "Generic (PLEG): container finished" podID="c751727a-5224-4b96-9a94-2430a78bd46a" containerID="a9a0706021e78655a789b05137af810a13ae9f7da6cb867a78cc20df28f4b96f" exitCode=0 Oct 30 21:38:57.549390 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.549321 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04ca9a6-p8zg4" event={"ID":"c751727a-5224-4b96-9a94-2430a78bd46a","Type":"ContainerDied","Data":"a9a0706021e78655a789b05137af810a13ae9f7da6cb867a78cc20df28f4b96f"} Oct 30 21:38:57.550919 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.550896 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e644c29-jb62q" event={"ID":"56d5db44-d8e7-4c47-b9b0-1bc2d52c891b","Type":"ContainerStarted","Data":"ed83303962d68fdfd6ab9706012393cc179e30eb363a8cf55edab64c55343355"} Oct 30 21:38:57.550987 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.550937 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e644c29-jb62q" event={"ID":"56d5db44-d8e7-4c47-b9b0-1bc2d52c891b","Type":"ContainerStarted","Data":"c2e665ee94efaeb5830160daa8318e36954c2770821a886fdf5a8b358620053c"} Oct 30 21:38:57.560777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.552778 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3de97-w4wjs" event={"ID":"16e1bc47-ed65-41be-acf1-bfb2b33ab26f","Type":"ContainerStarted","Data":"d64846f8c076d4407cad00c467df8fe7113aec79f61d606b58b544245ecd6869"} Oct 30 21:38:57.623520 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.623476 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e644c29-jb62q" podStartSLOduration=2.623462101 podStartE2EDuration="2.623462101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:57.621888349 +0000 UTC m=+606.217705035" watchObservedRunningTime="2026-10-30 21:38:57.623462101 +0000 UTC m=+606.219278758" Oct 30 21:38:57.709798 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.709748 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea80e817-pg2sx"] Oct 30 21:38:57.747716 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:38:57.747681 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d726c8c_7e7f_4b2b_ad3f_8c5ccdbdc59e.slice/crio-de2378c362fd37c7079f45e10cee0ca4b044ba9a3f5db95560d2444a3db90742 WatchSource:0}: Error finding container de2378c362fd37c7079f45e10cee0ca4b044ba9a3f5db95560d2444a3db90742: Status 404 returned error can't find the container with id de2378c362fd37c7079f45e10cee0ca4b044ba9a3f5db95560d2444a3db90742 Oct 30 21:38:57.808962 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.808927 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:38:57.809116 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.808989 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" podUID="1fcdad2f-d069-41a7-ace4-49d0ba4abfaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 21:38:57.930834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:57.930727 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:58.064777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:58.061084 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:38:58.064777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:58.061157 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" podUID="24b939ed-32e5-407e-a919-c0395d914dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:38:58.609526 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:58.609492 5801 generic.go:356] "Generic (PLEG): container finished" podID="56d5db44-d8e7-4c47-b9b0-1bc2d52c891b" containerID="ed83303962d68fdfd6ab9706012393cc179e30eb363a8cf55edab64c55343355" exitCode=0 Oct 30 21:38:58.609698 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:58.609568 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e644c29-jb62q" event={"ID":"56d5db44-d8e7-4c47-b9b0-1bc2d52c891b","Type":"ContainerDied","Data":"ed83303962d68fdfd6ab9706012393cc179e30eb363a8cf55edab64c55343355"} Oct 30 21:38:58.611500 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:58.611468 5801 generic.go:356] "Generic (PLEG): container finished" podID="16e1bc47-ed65-41be-acf1-bfb2b33ab26f" containerID="cba6e040e21aaccf56a9971017446ca8dee473191ae2cc7d4f08dda67f1b67c1" exitCode=0 Oct 30 21:38:58.611627 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:58.611558 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3de97-w4wjs" event={"ID":"16e1bc47-ed65-41be-acf1-bfb2b33ab26f","Type":"ContainerDied","Data":"cba6e040e21aaccf56a9971017446ca8dee473191ae2cc7d4f08dda67f1b67c1"} Oct 30 21:38:58.613536 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:58.613515 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea80e817-pg2sx" event={"ID":"9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e","Type":"ContainerStarted","Data":"832c16801e187f8e5eab21032594f4ec3b416e10ab0c759d97bf22417d9066f5"} Oct 30 21:38:58.613612 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:58.613550 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea80e817-pg2sx" event={"ID":"9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e","Type":"ContainerStarted","Data":"de2378c362fd37c7079f45e10cee0ca4b044ba9a3f5db95560d2444a3db90742"} Oct 30 21:38:58.647705 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:58.647659 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea80e817-pg2sx" podStartSLOduration=2.64764451 podStartE2EDuration="2.64764451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:58.645062456 +0000 UTC m=+607.240879135" watchObservedRunningTime="2026-10-30 21:38:58.64764451 +0000 UTC m=+607.243461170" Oct 30 21:38:58.740497 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:58.740470 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:58.843777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:58.834043 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:38:58.843777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:58.834105 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" podUID="1fcdad2f-d069-41a7-ace4-49d0ba4abfaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 21:38:58.947068 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:58.947044 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04ca9a6-p8zg4" Oct 30 21:38:59.036391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:59.035734 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c751727a-5224-4b96-9a94-2430a78bd46a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c751727a-5224-4b96-9a94-2430a78bd46a-maas-api-xks-ca\") pod \"c751727a-5224-4b96-9a94-2430a78bd46a\" (UID: \"c751727a-5224-4b96-9a94-2430a78bd46a\") " Oct 30 21:38:59.036391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:59.035838 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c751727a-5224-4b96-9a94-2430a78bd46a-kube-api-access-fzwfh\" (UniqueName: \"kubernetes.io/projected/c751727a-5224-4b96-9a94-2430a78bd46a-kube-api-access-fzwfh\") pod \"c751727a-5224-4b96-9a94-2430a78bd46a\" (UID: \"c751727a-5224-4b96-9a94-2430a78bd46a\") " Oct 30 21:38:59.036391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:59.035976 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c751727a-5224-4b96-9a94-2430a78bd46a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c751727a-5224-4b96-9a94-2430a78bd46a-maas-api-openshift-ca\") pod \"c751727a-5224-4b96-9a94-2430a78bd46a\" (UID: \"c751727a-5224-4b96-9a94-2430a78bd46a\") " Oct 30 21:38:59.036641 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:59.036557 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c751727a-5224-4b96-9a94-2430a78bd46a-maas-api-openshift-ca" pod "c751727a-5224-4b96-9a94-2430a78bd46a" (UID: "c751727a-5224-4b96-9a94-2430a78bd46a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:59.037998 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:59.037957 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c751727a-5224-4b96-9a94-2430a78bd46a-maas-api-xks-ca" pod "c751727a-5224-4b96-9a94-2430a78bd46a" (UID: "c751727a-5224-4b96-9a94-2430a78bd46a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:59.038462 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:59.038437 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c751727a-5224-4b96-9a94-2430a78bd46a-kube-api-access-fzwfh" pod "c751727a-5224-4b96-9a94-2430a78bd46a" (UID: "c751727a-5224-4b96-9a94-2430a78bd46a"). InnerVolumeSpecName "kube-api-access-fzwfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:59.103555 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:59.103516 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:38:59.103738 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:59.103574 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" podUID="24b939ed-32e5-407e-a919-c0395d914dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:38:59.137471 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:59.137440 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c751727a-5224-4b96-9a94-2430a78bd46a-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:38:59.137471 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:59.137466 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzwfh\" (UniqueName: \"kubernetes.io/projected/c751727a-5224-4b96-9a94-2430a78bd46a-kube-api-access-fzwfh\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:38:59.137471 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:59.137475 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c751727a-5224-4b96-9a94-2430a78bd46a-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:38:59.528392 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:59.528359 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:59.629594 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:59.627882 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04ca9a6-p8zg4" Oct 30 21:38:59.629594 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:59.627889 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04ca9a6-p8zg4" event={"ID":"c751727a-5224-4b96-9a94-2430a78bd46a","Type":"ContainerDied","Data":"c1a92f299a11f3279147349f071850ab6ba6bc77adda7335519807b0e9bf3498"} Oct 30 21:38:59.629594 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:59.628191 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1a92f299a11f3279147349f071850ab6ba6bc77adda7335519807b0e9bf3498" Oct 30 21:38:59.629909 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:59.629660 5801 generic.go:356] "Generic (PLEG): container finished" podID="9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e" containerID="832c16801e187f8e5eab21032594f4ec3b416e10ab0c759d97bf22417d9066f5" exitCode=0 Oct 30 21:38:59.629978 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:59.629853 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea80e817-pg2sx" event={"ID":"9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e","Type":"ContainerDied","Data":"832c16801e187f8e5eab21032594f4ec3b416e10ab0c759d97bf22417d9066f5"} Oct 30 21:38:59.809398 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:59.809363 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:38:59.809577 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:59.809432 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" podUID="1fcdad2f-d069-41a7-ace4-49d0ba4abfaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 21:38:59.945584 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:59.945566 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3de97-w4wjs" Oct 30 21:38:59.947831 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:38:59.947812 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e644c29-jb62q" Oct 30 21:39:00.048315 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.048287 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16e1bc47-ed65-41be-acf1-bfb2b33ab26f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e1bc47-ed65-41be-acf1-bfb2b33ab26f-maas-api-xks-ca\") pod \"16e1bc47-ed65-41be-acf1-bfb2b33ab26f\" (UID: \"16e1bc47-ed65-41be-acf1-bfb2b33ab26f\") " Oct 30 21:39:00.048466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.048346 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e1bc47-ed65-41be-acf1-bfb2b33ab26f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e1bc47-ed65-41be-acf1-bfb2b33ab26f-maas-api-openshift-ca\") pod \"16e1bc47-ed65-41be-acf1-bfb2b33ab26f\" (UID: \"16e1bc47-ed65-41be-acf1-bfb2b33ab26f\") " Oct 30 21:39:00.048466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.048372 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d5db44-d8e7-4c47-b9b0-1bc2d52c891b-kube-api-access-wr77k\" (UniqueName: \"kubernetes.io/projected/56d5db44-d8e7-4c47-b9b0-1bc2d52c891b-kube-api-access-wr77k\") pod \"56d5db44-d8e7-4c47-b9b0-1bc2d52c891b\" (UID: \"56d5db44-d8e7-4c47-b9b0-1bc2d52c891b\") " Oct 30 21:39:00.048466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.048428 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56d5db44-d8e7-4c47-b9b0-1bc2d52c891b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d5db44-d8e7-4c47-b9b0-1bc2d52c891b-maas-api-xks-ca\") pod \"56d5db44-d8e7-4c47-b9b0-1bc2d52c891b\" (UID: \"56d5db44-d8e7-4c47-b9b0-1bc2d52c891b\") " Oct 30 21:39:00.048466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.048451 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56d5db44-d8e7-4c47-b9b0-1bc2d52c891b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d5db44-d8e7-4c47-b9b0-1bc2d52c891b-maas-api-openshift-ca\") pod \"56d5db44-d8e7-4c47-b9b0-1bc2d52c891b\" (UID: \"56d5db44-d8e7-4c47-b9b0-1bc2d52c891b\") " Oct 30 21:39:00.048589 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.048475 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e1bc47-ed65-41be-acf1-bfb2b33ab26f-kube-api-access-p8rqf\" (UniqueName: \"kubernetes.io/projected/16e1bc47-ed65-41be-acf1-bfb2b33ab26f-kube-api-access-p8rqf\") pod \"16e1bc47-ed65-41be-acf1-bfb2b33ab26f\" (UID: \"16e1bc47-ed65-41be-acf1-bfb2b33ab26f\") " Oct 30 21:39:00.048708 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.048683 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e1bc47-ed65-41be-acf1-bfb2b33ab26f-maas-api-openshift-ca" pod "16e1bc47-ed65-41be-acf1-bfb2b33ab26f" (UID: "16e1bc47-ed65-41be-acf1-bfb2b33ab26f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:00.048989 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.048831 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e1bc47-ed65-41be-acf1-bfb2b33ab26f-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:00.049221 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.049098 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d5db44-d8e7-4c47-b9b0-1bc2d52c891b-maas-api-openshift-ca" pod "56d5db44-d8e7-4c47-b9b0-1bc2d52c891b" (UID: "56d5db44-d8e7-4c47-b9b0-1bc2d52c891b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:00.050996 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.050966 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e1bc47-ed65-41be-acf1-bfb2b33ab26f-kube-api-access-p8rqf" pod "16e1bc47-ed65-41be-acf1-bfb2b33ab26f" (UID: "16e1bc47-ed65-41be-acf1-bfb2b33ab26f"). InnerVolumeSpecName "kube-api-access-p8rqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:00.051090 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.051068 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d5db44-d8e7-4c47-b9b0-1bc2d52c891b-kube-api-access-wr77k" pod "56d5db44-d8e7-4c47-b9b0-1bc2d52c891b" (UID: "56d5db44-d8e7-4c47-b9b0-1bc2d52c891b"). InnerVolumeSpecName "kube-api-access-wr77k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:00.051279 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.051264 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e1bc47-ed65-41be-acf1-bfb2b33ab26f-maas-api-xks-ca" pod "16e1bc47-ed65-41be-acf1-bfb2b33ab26f" (UID: "16e1bc47-ed65-41be-acf1-bfb2b33ab26f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:00.051483 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.051469 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56d5db44-d8e7-4c47-b9b0-1bc2d52c891b-maas-api-xks-ca" pod "56d5db44-d8e7-4c47-b9b0-1bc2d52c891b" (UID: "56d5db44-d8e7-4c47-b9b0-1bc2d52c891b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:00.108465 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.093885 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:39:00.108465 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.093948 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" podUID="24b939ed-32e5-407e-a919-c0395d914dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:39:00.150158 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.150126 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e1bc47-ed65-41be-acf1-bfb2b33ab26f-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:00.150158 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.150154 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr77k\" (UniqueName: \"kubernetes.io/projected/56d5db44-d8e7-4c47-b9b0-1bc2d52c891b-kube-api-access-wr77k\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:00.150158 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.150164 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d5db44-d8e7-4c47-b9b0-1bc2d52c891b-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:00.150364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.150174 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d5db44-d8e7-4c47-b9b0-1bc2d52c891b-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:00.150364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.150183 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8rqf\" (UniqueName: \"kubernetes.io/projected/16e1bc47-ed65-41be-acf1-bfb2b33ab26f-kube-api-access-p8rqf\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:00.240627 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.240593 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:00.303961 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:39:00.303922 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:39:00.638098 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.638064 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e644c29-jb62q" Oct 30 21:39:00.638098 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.638059 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e644c29-jb62q" event={"ID":"56d5db44-d8e7-4c47-b9b0-1bc2d52c891b","Type":"ContainerDied","Data":"c2e665ee94efaeb5830160daa8318e36954c2770821a886fdf5a8b358620053c"} Oct 30 21:39:00.638383 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.638153 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2e665ee94efaeb5830160daa8318e36954c2770821a886fdf5a8b358620053c" Oct 30 21:39:00.639742 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.639722 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3de97-w4wjs" Oct 30 21:39:00.639944 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.639744 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c3de97-w4wjs" event={"ID":"16e1bc47-ed65-41be-acf1-bfb2b33ab26f","Type":"ContainerDied","Data":"d64846f8c076d4407cad00c467df8fe7113aec79f61d606b58b544245ecd6869"} Oct 30 21:39:00.640078 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.640064 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d64846f8c076d4407cad00c467df8fe7113aec79f61d606b58b544245ecd6869" Oct 30 21:39:00.864794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.861509 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea80e817-pg2sx" Oct 30 21:39:00.995827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.995023 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e-maas-api-xks-ca\") pod \"9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e\" (UID: \"9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e\") " Oct 30 21:39:00.995827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.995551 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e-kube-api-access-67t26\" (UniqueName: \"kubernetes.io/projected/9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e-kube-api-access-67t26\") pod \"9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e\" (UID: \"9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e\") " Oct 30 21:39:00.996358 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.996038 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e-maas-api-openshift-ca\") pod \"9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e\" (UID: \"9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e\") " Oct 30 21:39:00.998170 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.996482 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e-maas-api-openshift-ca" pod "9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e" (UID: "9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:00.998170 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.996602 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:01.001777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:00.999400 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e-kube-api-access-67t26" pod "9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e" (UID: "9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e"). InnerVolumeSpecName "kube-api-access-67t26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:01.003171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:01.003132 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e-maas-api-xks-ca" pod "9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e" (UID: "9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:01.015460 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:01.015365 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:01.062275 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:01.061390 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:39:01.062275 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:01.061443 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" podUID="24b939ed-32e5-407e-a919-c0395d914dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:39:01.098450 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:01.098414 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:01.098450 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:01.098448 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67t26\" (UniqueName: \"kubernetes.io/projected/9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e-kube-api-access-67t26\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:01.171791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:01.168125 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:39:01.171791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:01.169882 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:39:01.172990 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:01.172962 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:39:01.652481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:01.652450 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea80e817-pg2sx" Oct 30 21:39:01.656843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:01.656815 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea80e817-pg2sx" event={"ID":"9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e","Type":"ContainerDied","Data":"de2378c362fd37c7079f45e10cee0ca4b044ba9a3f5db95560d2444a3db90742"} Oct 30 21:39:01.657057 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:01.657037 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de2378c362fd37c7079f45e10cee0ca4b044ba9a3f5db95560d2444a3db90742" Oct 30 21:39:01.915353 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:01.915266 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:02.303694 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:02.303658 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:39:02.304499 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:02.304475 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:39:02.305316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:02.305296 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:39:02.738488 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:02.738394 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:03.432934 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:03.432901 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:04.170521 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:04.170463 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:04.951142 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:04.951089 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:04.986357 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:04.986321 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-762d372d-z4w6z"] Oct 30 21:39:04.987142 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:04.987115 5801 cpu_manager.go:414] "containerMap: removing container" podUID="56d5db44-d8e7-4c47-b9b0-1bc2d52c891b" containerName="revoke-keys" Oct 30 21:39:04.987142 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:04.987141 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d5db44-d8e7-4c47-b9b0-1bc2d52c891b" containerName="revoke-keys" Oct 30 21:39:04.987331 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:04.987161 5801 cpu_manager.go:414] "containerMap: removing container" podUID="c751727a-5224-4b96-9a94-2430a78bd46a" containerName="revoke-keys" Oct 30 21:39:04.987331 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:04.987168 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c751727a-5224-4b96-9a94-2430a78bd46a" containerName="revoke-keys" Oct 30 21:39:04.987331 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:04.987181 5801 cpu_manager.go:414] "containerMap: removing container" podUID="16e1bc47-ed65-41be-acf1-bfb2b33ab26f" containerName="revoke-keys" Oct 30 21:39:04.987331 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:04.987189 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e1bc47-ed65-41be-acf1-bfb2b33ab26f" containerName="revoke-keys" Oct 30 21:39:04.987331 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:04.987210 5801 cpu_manager.go:414] "containerMap: removing container" podUID="9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e" containerName="revoke-keys" Oct 30 21:39:04.987331 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:04.987216 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e" containerName="revoke-keys" Oct 30 21:39:04.994145 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:04.987554 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d5db44-d8e7-4c47-b9b0-1bc2d52c891b" containerName="revoke-keys" Oct 30 21:39:04.994145 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:04.987575 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="c751727a-5224-4b96-9a94-2430a78bd46a" containerName="revoke-keys" Oct 30 21:39:04.994145 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:04.987591 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e1bc47-ed65-41be-acf1-bfb2b33ab26f" containerName="revoke-keys" Oct 30 21:39:04.994145 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:04.987603 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e" containerName="revoke-keys" Oct 30 21:39:04.998048 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:04.998020 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762d372d-z4w6z" Oct 30 21:39:05.000329 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.000288 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-762d372d-z4w6z"] Oct 30 21:39:05.003785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.003764 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:39:05.004013 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.003991 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:39:05.004100 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.004070 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:39:05.004100 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.004093 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:39:05.064018 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.063984 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5899e00-98f2-4184-a863-4cafff2ae618-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-762d372d-z4w6z\" (UID: \"a5899e00-98f2-4184-a863-4cafff2ae618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762d372d-z4w6z" Oct 30 21:39:05.064168 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.064056 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5899e00-98f2-4184-a863-4cafff2ae618-maas-api-xks-ca\") pod \"maas-api-revoke-sub-762d372d-z4w6z\" (UID: \"a5899e00-98f2-4184-a863-4cafff2ae618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762d372d-z4w6z" Oct 30 21:39:05.064168 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.064122 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvk7k\" (UniqueName: \"kubernetes.io/projected/a5899e00-98f2-4184-a863-4cafff2ae618-kube-api-access-xvk7k\") pod \"maas-api-revoke-sub-762d372d-z4w6z\" (UID: \"a5899e00-98f2-4184-a863-4cafff2ae618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762d372d-z4w6z" Oct 30 21:39:05.165015 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.164983 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5899e00-98f2-4184-a863-4cafff2ae618-maas-api-xks-ca\") pod \"maas-api-revoke-sub-762d372d-z4w6z\" (UID: \"a5899e00-98f2-4184-a863-4cafff2ae618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762d372d-z4w6z" Oct 30 21:39:05.165172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.165031 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvk7k\" (UniqueName: \"kubernetes.io/projected/a5899e00-98f2-4184-a863-4cafff2ae618-kube-api-access-xvk7k\") pod \"maas-api-revoke-sub-762d372d-z4w6z\" (UID: \"a5899e00-98f2-4184-a863-4cafff2ae618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762d372d-z4w6z" Oct 30 21:39:05.165209 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.165184 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5899e00-98f2-4184-a863-4cafff2ae618-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-762d372d-z4w6z\" (UID: \"a5899e00-98f2-4184-a863-4cafff2ae618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762d372d-z4w6z" Oct 30 21:39:05.165834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.165810 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5899e00-98f2-4184-a863-4cafff2ae618-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-762d372d-z4w6z\" (UID: \"a5899e00-98f2-4184-a863-4cafff2ae618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762d372d-z4w6z" Oct 30 21:39:05.167525 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.167498 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5899e00-98f2-4184-a863-4cafff2ae618-maas-api-xks-ca\") pod \"maas-api-revoke-sub-762d372d-z4w6z\" (UID: \"a5899e00-98f2-4184-a863-4cafff2ae618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762d372d-z4w6z" Oct 30 21:39:05.177637 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.177608 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvk7k\" (UniqueName: \"kubernetes.io/projected/a5899e00-98f2-4184-a863-4cafff2ae618-kube-api-access-xvk7k\") pod \"maas-api-revoke-sub-762d372d-z4w6z\" (UID: \"a5899e00-98f2-4184-a863-4cafff2ae618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762d372d-z4w6z" Oct 30 21:39:05.317244 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.317214 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762d372d-z4w6z" Oct 30 21:39:05.343896 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.343869 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b2286e6-5v87x"] Oct 30 21:39:05.348790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.348767 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b2286e6-5v87x" Oct 30 21:39:05.362100 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.359825 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b2286e6-5v87x"] Oct 30 21:39:05.472807 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.472771 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26fc0e5-bd68-4d01-9fb8-9c16c1971b74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b2286e6-5v87x\" (UID: \"c26fc0e5-bd68-4d01-9fb8-9c16c1971b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b2286e6-5v87x" Oct 30 21:39:05.472953 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.472910 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26fc0e5-bd68-4d01-9fb8-9c16c1971b74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b2286e6-5v87x\" (UID: \"c26fc0e5-bd68-4d01-9fb8-9c16c1971b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b2286e6-5v87x" Oct 30 21:39:05.472953 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.472948 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68djv\" (UniqueName: \"kubernetes.io/projected/c26fc0e5-bd68-4d01-9fb8-9c16c1971b74-kube-api-access-68djv\") pod \"maas-api-revoke-sub-9b2286e6-5v87x\" (UID: \"c26fc0e5-bd68-4d01-9fb8-9c16c1971b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b2286e6-5v87x" Oct 30 21:39:05.573964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.573873 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26fc0e5-bd68-4d01-9fb8-9c16c1971b74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b2286e6-5v87x\" (UID: \"c26fc0e5-bd68-4d01-9fb8-9c16c1971b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b2286e6-5v87x" Oct 30 21:39:05.573964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.573928 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68djv\" (UniqueName: \"kubernetes.io/projected/c26fc0e5-bd68-4d01-9fb8-9c16c1971b74-kube-api-access-68djv\") pod \"maas-api-revoke-sub-9b2286e6-5v87x\" (UID: \"c26fc0e5-bd68-4d01-9fb8-9c16c1971b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b2286e6-5v87x" Oct 30 21:39:05.574197 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.574034 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26fc0e5-bd68-4d01-9fb8-9c16c1971b74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b2286e6-5v87x\" (UID: \"c26fc0e5-bd68-4d01-9fb8-9c16c1971b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b2286e6-5v87x" Oct 30 21:39:05.574704 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.574677 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26fc0e5-bd68-4d01-9fb8-9c16c1971b74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b2286e6-5v87x\" (UID: \"c26fc0e5-bd68-4d01-9fb8-9c16c1971b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b2286e6-5v87x" Oct 30 21:39:05.576239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.576217 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26fc0e5-bd68-4d01-9fb8-9c16c1971b74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b2286e6-5v87x\" (UID: \"c26fc0e5-bd68-4d01-9fb8-9c16c1971b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b2286e6-5v87x" Oct 30 21:39:05.586278 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.586250 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68djv\" (UniqueName: \"kubernetes.io/projected/c26fc0e5-bd68-4d01-9fb8-9c16c1971b74-kube-api-access-68djv\") pod \"maas-api-revoke-sub-9b2286e6-5v87x\" (UID: \"c26fc0e5-bd68-4d01-9fb8-9c16c1971b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b2286e6-5v87x" Oct 30 21:39:05.615159 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.615135 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-762d372d-z4w6z"] Oct 30 21:39:05.616862 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:39:05.616831 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5899e00_98f2_4184_a863_4cafff2ae618.slice/crio-d5c9cb16cc3d61a6311a1afdec2bb0ce55f43cc0a865864c4f7452a0187a7234 WatchSource:0}: Error finding container d5c9cb16cc3d61a6311a1afdec2bb0ce55f43cc0a865864c4f7452a0187a7234: Status 404 returned error can't find the container with id d5c9cb16cc3d61a6311a1afdec2bb0ce55f43cc0a865864c4f7452a0187a7234 Oct 30 21:39:05.663554 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.663526 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:05.675130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.675107 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b2286e6-5v87x" Oct 30 21:39:05.676736 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.676714 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762d372d-z4w6z" event={"ID":"a5899e00-98f2-4184-a863-4cafff2ae618","Type":"ContainerStarted","Data":"d5c9cb16cc3d61a6311a1afdec2bb0ce55f43cc0a865864c4f7452a0187a7234"} Oct 30 21:39:05.973042 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:05.973014 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b2286e6-5v87x"] Oct 30 21:39:05.975057 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:39:05.975032 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26fc0e5_bd68_4d01_9fb8_9c16c1971b74.slice/crio-ca72c69df54ed94d82feff9de102f6b1e4d3739f8986558306c8cf9d319c2652 WatchSource:0}: Error finding container ca72c69df54ed94d82feff9de102f6b1e4d3739f8986558306c8cf9d319c2652: Status 404 returned error can't find the container with id ca72c69df54ed94d82feff9de102f6b1e4d3739f8986558306c8cf9d319c2652 Oct 30 21:39:06.415384 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:06.415352 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:06.684342 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:06.684311 5801 generic.go:356] "Generic (PLEG): container finished" podID="a5899e00-98f2-4184-a863-4cafff2ae618" containerID="519b35bc928c60457e4d5062c94a1d7c82f31ad776a56b508c1bd1701c1bce57" exitCode=0 Oct 30 21:39:06.684551 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:06.684393 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762d372d-z4w6z" event={"ID":"a5899e00-98f2-4184-a863-4cafff2ae618","Type":"ContainerDied","Data":"519b35bc928c60457e4d5062c94a1d7c82f31ad776a56b508c1bd1701c1bce57"} Oct 30 21:39:06.686095 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:06.686073 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b2286e6-5v87x" event={"ID":"c26fc0e5-bd68-4d01-9fb8-9c16c1971b74","Type":"ContainerStarted","Data":"97ef059f678c1a11cb8d5e575356942672a321d56bf0a2b659ae6d14be6382bb"} Oct 30 21:39:06.686195 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:06.686111 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b2286e6-5v87x" event={"ID":"c26fc0e5-bd68-4d01-9fb8-9c16c1971b74","Type":"ContainerStarted","Data":"ca72c69df54ed94d82feff9de102f6b1e4d3739f8986558306c8cf9d319c2652"} Oct 30 21:39:06.728644 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:06.728542 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b2286e6-5v87x" podStartSLOduration=1.728529972 podStartE2EDuration="1.728529972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:06.72530218 +0000 UTC m=+615.321118864" watchObservedRunningTime="2026-10-30 21:39:06.728529972 +0000 UTC m=+615.324346632" Oct 30 21:39:06.753011 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:06.752981 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed7d1d1-9xnnw"] Oct 30 21:39:06.756665 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:06.756648 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7d1d1-9xnnw" Oct 30 21:39:06.766248 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:06.766221 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed7d1d1-9xnnw"] Oct 30 21:39:06.888225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:06.888193 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxhw5\" (UniqueName: \"kubernetes.io/projected/f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f-kube-api-access-jxhw5\") pod \"maas-api-revoke-sub-9ed7d1d1-9xnnw\" (UID: \"f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7d1d1-9xnnw" Oct 30 21:39:06.888442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:06.888417 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ed7d1d1-9xnnw\" (UID: \"f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7d1d1-9xnnw" Oct 30 21:39:06.888640 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:06.888536 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ed7d1d1-9xnnw\" (UID: \"f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7d1d1-9xnnw" Oct 30 21:39:06.989404 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:06.989309 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ed7d1d1-9xnnw\" (UID: \"f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7d1d1-9xnnw" Oct 30 21:39:06.989404 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:06.989387 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ed7d1d1-9xnnw\" (UID: \"f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7d1d1-9xnnw" Oct 30 21:39:06.989932 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:06.989467 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxhw5\" (UniqueName: \"kubernetes.io/projected/f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f-kube-api-access-jxhw5\") pod \"maas-api-revoke-sub-9ed7d1d1-9xnnw\" (UID: \"f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7d1d1-9xnnw" Oct 30 21:39:06.990094 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:06.990073 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ed7d1d1-9xnnw\" (UID: \"f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7d1d1-9xnnw" Oct 30 21:39:06.991656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:06.991638 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ed7d1d1-9xnnw\" (UID: \"f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7d1d1-9xnnw" Oct 30 21:39:07.002746 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:07.002721 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxhw5\" (UniqueName: \"kubernetes.io/projected/f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f-kube-api-access-jxhw5\") pod \"maas-api-revoke-sub-9ed7d1d1-9xnnw\" (UID: \"f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7d1d1-9xnnw" Oct 30 21:39:07.071299 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:07.071268 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7d1d1-9xnnw" Oct 30 21:39:07.223022 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:07.222981 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:07.348600 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:07.348572 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed7d1d1-9xnnw"] Oct 30 21:39:07.349692 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:39:07.349654 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6dd41de_7a1e_4d5b_ae2c_0500b7ad116f.slice/crio-ddcf39eda13cba9a19a68b08e491b0476fb0cea2df59c3f5cd2a5fea6efaa03c WatchSource:0}: Error finding container ddcf39eda13cba9a19a68b08e491b0476fb0cea2df59c3f5cd2a5fea6efaa03c: Status 404 returned error can't find the container with id ddcf39eda13cba9a19a68b08e491b0476fb0cea2df59c3f5cd2a5fea6efaa03c Oct 30 21:39:07.693331 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:07.693295 5801 generic.go:356] "Generic (PLEG): container finished" podID="c26fc0e5-bd68-4d01-9fb8-9c16c1971b74" containerID="97ef059f678c1a11cb8d5e575356942672a321d56bf0a2b659ae6d14be6382bb" exitCode=0 Oct 30 21:39:07.693462 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:07.693337 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b2286e6-5v87x" event={"ID":"c26fc0e5-bd68-4d01-9fb8-9c16c1971b74","Type":"ContainerDied","Data":"97ef059f678c1a11cb8d5e575356942672a321d56bf0a2b659ae6d14be6382bb"} Oct 30 21:39:07.694944 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:07.694920 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7d1d1-9xnnw" event={"ID":"f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f","Type":"ContainerStarted","Data":"49605a76a8ac69fff2e2179c87e7a42f9cc11788246cd8ef4377083b23dd2dcb"} Oct 30 21:39:07.695050 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:07.694957 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7d1d1-9xnnw" event={"ID":"f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f","Type":"ContainerStarted","Data":"ddcf39eda13cba9a19a68b08e491b0476fb0cea2df59c3f5cd2a5fea6efaa03c"} Oct 30 21:39:07.796688 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:07.796634 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7d1d1-9xnnw" podStartSLOduration=1.7966183770000002 podStartE2EDuration="1.796618377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:07.795098291 +0000 UTC m=+616.390914950" watchObservedRunningTime="2026-10-30 21:39:07.796618377 +0000 UTC m=+616.392435060" Oct 30 21:39:07.843107 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:07.843075 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762d372d-z4w6z" Oct 30 21:39:08.002041 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.001953 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5899e00-98f2-4184-a863-4cafff2ae618-kube-api-access-xvk7k\" (UniqueName: \"kubernetes.io/projected/a5899e00-98f2-4184-a863-4cafff2ae618-kube-api-access-xvk7k\") pod \"a5899e00-98f2-4184-a863-4cafff2ae618\" (UID: \"a5899e00-98f2-4184-a863-4cafff2ae618\") " Oct 30 21:39:08.002041 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.002027 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5899e00-98f2-4184-a863-4cafff2ae618-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5899e00-98f2-4184-a863-4cafff2ae618-maas-api-openshift-ca\") pod \"a5899e00-98f2-4184-a863-4cafff2ae618\" (UID: \"a5899e00-98f2-4184-a863-4cafff2ae618\") " Oct 30 21:39:08.002476 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.002060 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5899e00-98f2-4184-a863-4cafff2ae618-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5899e00-98f2-4184-a863-4cafff2ae618-maas-api-xks-ca\") pod \"a5899e00-98f2-4184-a863-4cafff2ae618\" (UID: \"a5899e00-98f2-4184-a863-4cafff2ae618\") " Oct 30 21:39:08.002476 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.002408 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5899e00-98f2-4184-a863-4cafff2ae618-maas-api-openshift-ca" pod "a5899e00-98f2-4184-a863-4cafff2ae618" (UID: "a5899e00-98f2-4184-a863-4cafff2ae618"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:08.002608 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.002593 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5899e00-98f2-4184-a863-4cafff2ae618-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:08.004162 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.004134 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5899e00-98f2-4184-a863-4cafff2ae618-maas-api-xks-ca" pod "a5899e00-98f2-4184-a863-4cafff2ae618" (UID: "a5899e00-98f2-4184-a863-4cafff2ae618"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:08.004266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.004205 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5899e00-98f2-4184-a863-4cafff2ae618-kube-api-access-xvk7k" pod "a5899e00-98f2-4184-a863-4cafff2ae618" (UID: "a5899e00-98f2-4184-a863-4cafff2ae618"). InnerVolumeSpecName "kube-api-access-xvk7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:08.050493 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.050457 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8991907d-wxl9z"] Oct 30 21:39:08.051019 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.051004 5801 cpu_manager.go:414] "containerMap: removing container" podUID="a5899e00-98f2-4184-a863-4cafff2ae618" containerName="revoke-keys" Oct 30 21:39:08.051062 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.051020 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5899e00-98f2-4184-a863-4cafff2ae618" containerName="revoke-keys" Oct 30 21:39:08.051112 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.051102 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5899e00-98f2-4184-a863-4cafff2ae618" containerName="revoke-keys" Oct 30 21:39:08.054472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.054456 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8991907d-wxl9z" Oct 30 21:39:08.064616 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.064586 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8991907d-wxl9z"] Oct 30 21:39:08.104030 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.103996 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvk7k\" (UniqueName: \"kubernetes.io/projected/a5899e00-98f2-4184-a863-4cafff2ae618-kube-api-access-xvk7k\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:08.104030 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.104026 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5899e00-98f2-4184-a863-4cafff2ae618-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:08.164079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.164040 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:08.205202 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.205163 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8d7q\" (UniqueName: \"kubernetes.io/projected/d067f135-fcc0-4578-bb69-6fb5a6ecf49e-kube-api-access-d8d7q\") pod \"maas-api-revoke-sub-8991907d-wxl9z\" (UID: \"d067f135-fcc0-4578-bb69-6fb5a6ecf49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8991907d-wxl9z" Oct 30 21:39:08.205427 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.205228 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d067f135-fcc0-4578-bb69-6fb5a6ecf49e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8991907d-wxl9z\" (UID: \"d067f135-fcc0-4578-bb69-6fb5a6ecf49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8991907d-wxl9z" Oct 30 21:39:08.205427 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.205272 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d067f135-fcc0-4578-bb69-6fb5a6ecf49e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8991907d-wxl9z\" (UID: \"d067f135-fcc0-4578-bb69-6fb5a6ecf49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8991907d-wxl9z" Oct 30 21:39:08.306491 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.306454 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8d7q\" (UniqueName: \"kubernetes.io/projected/d067f135-fcc0-4578-bb69-6fb5a6ecf49e-kube-api-access-d8d7q\") pod \"maas-api-revoke-sub-8991907d-wxl9z\" (UID: \"d067f135-fcc0-4578-bb69-6fb5a6ecf49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8991907d-wxl9z" Oct 30 21:39:08.306664 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.306504 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d067f135-fcc0-4578-bb69-6fb5a6ecf49e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8991907d-wxl9z\" (UID: \"d067f135-fcc0-4578-bb69-6fb5a6ecf49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8991907d-wxl9z" Oct 30 21:39:08.306664 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.306538 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d067f135-fcc0-4578-bb69-6fb5a6ecf49e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8991907d-wxl9z\" (UID: \"d067f135-fcc0-4578-bb69-6fb5a6ecf49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8991907d-wxl9z" Oct 30 21:39:08.307274 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.307254 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d067f135-fcc0-4578-bb69-6fb5a6ecf49e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8991907d-wxl9z\" (UID: \"d067f135-fcc0-4578-bb69-6fb5a6ecf49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8991907d-wxl9z" Oct 30 21:39:08.308847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.308828 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d067f135-fcc0-4578-bb69-6fb5a6ecf49e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8991907d-wxl9z\" (UID: \"d067f135-fcc0-4578-bb69-6fb5a6ecf49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8991907d-wxl9z" Oct 30 21:39:08.319306 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.319273 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8d7q\" (UniqueName: \"kubernetes.io/projected/d067f135-fcc0-4578-bb69-6fb5a6ecf49e-kube-api-access-d8d7q\") pod \"maas-api-revoke-sub-8991907d-wxl9z\" (UID: \"d067f135-fcc0-4578-bb69-6fb5a6ecf49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8991907d-wxl9z" Oct 30 21:39:08.368948 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.368911 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8991907d-wxl9z" Oct 30 21:39:08.653514 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.653476 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8991907d-wxl9z"] Oct 30 21:39:08.656028 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:39:08.656004 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd067f135_fcc0_4578_bb69_6fb5a6ecf49e.slice/crio-1a70a8ad2017b7e65d473fafb928fdc90810183968930fda2d256d48b5c98a39 WatchSource:0}: Error finding container 1a70a8ad2017b7e65d473fafb928fdc90810183968930fda2d256d48b5c98a39: Status 404 returned error can't find the container with id 1a70a8ad2017b7e65d473fafb928fdc90810183968930fda2d256d48b5c98a39 Oct 30 21:39:08.680213 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.680188 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e44bde8-l5x9w"] Oct 30 21:39:08.685206 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.685190 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44bde8-l5x9w" Oct 30 21:39:08.693040 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.693015 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e44bde8-l5x9w"] Oct 30 21:39:08.705932 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.705903 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8991907d-wxl9z" event={"ID":"d067f135-fcc0-4578-bb69-6fb5a6ecf49e","Type":"ContainerStarted","Data":"1a70a8ad2017b7e65d473fafb928fdc90810183968930fda2d256d48b5c98a39"} Oct 30 21:39:08.707919 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.707890 5801 generic.go:356] "Generic (PLEG): container finished" podID="f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f" containerID="49605a76a8ac69fff2e2179c87e7a42f9cc11788246cd8ef4377083b23dd2dcb" exitCode=0 Oct 30 21:39:08.708036 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.708016 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7d1d1-9xnnw" event={"ID":"f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f","Type":"ContainerDied","Data":"49605a76a8ac69fff2e2179c87e7a42f9cc11788246cd8ef4377083b23dd2dcb"} Oct 30 21:39:08.711206 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.711067 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762d372d-z4w6z" event={"ID":"a5899e00-98f2-4184-a863-4cafff2ae618","Type":"ContainerDied","Data":"d5c9cb16cc3d61a6311a1afdec2bb0ce55f43cc0a865864c4f7452a0187a7234"} Oct 30 21:39:08.711206 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.711120 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5c9cb16cc3d61a6311a1afdec2bb0ce55f43cc0a865864c4f7452a0187a7234" Oct 30 21:39:08.711363 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.711239 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762d372d-z4w6z" Oct 30 21:39:08.812708 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.812674 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c9fa99-b9a5-452c-b352-481a0cf4ec4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e44bde8-l5x9w\" (UID: \"44c9fa99-b9a5-452c-b352-481a0cf4ec4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44bde8-l5x9w" Oct 30 21:39:08.812916 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.812723 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbjkr\" (UniqueName: \"kubernetes.io/projected/44c9fa99-b9a5-452c-b352-481a0cf4ec4a-kube-api-access-wbjkr\") pod \"maas-api-revoke-sub-4e44bde8-l5x9w\" (UID: \"44c9fa99-b9a5-452c-b352-481a0cf4ec4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44bde8-l5x9w" Oct 30 21:39:08.813034 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.813018 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c9fa99-b9a5-452c-b352-481a0cf4ec4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e44bde8-l5x9w\" (UID: \"44c9fa99-b9a5-452c-b352-481a0cf4ec4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44bde8-l5x9w" Oct 30 21:39:08.851632 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.851605 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b2286e6-5v87x" Oct 30 21:39:08.914042 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.913957 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c9fa99-b9a5-452c-b352-481a0cf4ec4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e44bde8-l5x9w\" (UID: \"44c9fa99-b9a5-452c-b352-481a0cf4ec4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44bde8-l5x9w" Oct 30 21:39:08.914042 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.913998 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbjkr\" (UniqueName: \"kubernetes.io/projected/44c9fa99-b9a5-452c-b352-481a0cf4ec4a-kube-api-access-wbjkr\") pod \"maas-api-revoke-sub-4e44bde8-l5x9w\" (UID: \"44c9fa99-b9a5-452c-b352-481a0cf4ec4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44bde8-l5x9w" Oct 30 21:39:08.914280 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.914144 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c9fa99-b9a5-452c-b352-481a0cf4ec4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e44bde8-l5x9w\" (UID: \"44c9fa99-b9a5-452c-b352-481a0cf4ec4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44bde8-l5x9w" Oct 30 21:39:08.914610 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.914584 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c9fa99-b9a5-452c-b352-481a0cf4ec4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e44bde8-l5x9w\" (UID: \"44c9fa99-b9a5-452c-b352-481a0cf4ec4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44bde8-l5x9w" Oct 30 21:39:08.953519 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.953479 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c9fa99-b9a5-452c-b352-481a0cf4ec4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e44bde8-l5x9w\" (UID: \"44c9fa99-b9a5-452c-b352-481a0cf4ec4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44bde8-l5x9w" Oct 30 21:39:08.953675 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:08.953637 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbjkr\" (UniqueName: \"kubernetes.io/projected/44c9fa99-b9a5-452c-b352-481a0cf4ec4a-kube-api-access-wbjkr\") pod \"maas-api-revoke-sub-4e44bde8-l5x9w\" (UID: \"44c9fa99-b9a5-452c-b352-481a0cf4ec4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44bde8-l5x9w" Oct 30 21:39:09.012214 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.012180 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44bde8-l5x9w" Oct 30 21:39:09.015372 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.015315 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c26fc0e5-bd68-4d01-9fb8-9c16c1971b74-kube-api-access-68djv\" (UniqueName: \"kubernetes.io/projected/c26fc0e5-bd68-4d01-9fb8-9c16c1971b74-kube-api-access-68djv\") pod \"c26fc0e5-bd68-4d01-9fb8-9c16c1971b74\" (UID: \"c26fc0e5-bd68-4d01-9fb8-9c16c1971b74\") " Oct 30 21:39:09.015441 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.015427 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c26fc0e5-bd68-4d01-9fb8-9c16c1971b74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26fc0e5-bd68-4d01-9fb8-9c16c1971b74-maas-api-xks-ca\") pod \"c26fc0e5-bd68-4d01-9fb8-9c16c1971b74\" (UID: \"c26fc0e5-bd68-4d01-9fb8-9c16c1971b74\") " Oct 30 21:39:09.015488 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.015460 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c26fc0e5-bd68-4d01-9fb8-9c16c1971b74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26fc0e5-bd68-4d01-9fb8-9c16c1971b74-maas-api-openshift-ca\") pod \"c26fc0e5-bd68-4d01-9fb8-9c16c1971b74\" (UID: \"c26fc0e5-bd68-4d01-9fb8-9c16c1971b74\") " Oct 30 21:39:09.015926 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.015894 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26fc0e5-bd68-4d01-9fb8-9c16c1971b74-maas-api-openshift-ca" pod "c26fc0e5-bd68-4d01-9fb8-9c16c1971b74" (UID: "c26fc0e5-bd68-4d01-9fb8-9c16c1971b74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:09.017508 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.017483 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c26fc0e5-bd68-4d01-9fb8-9c16c1971b74-maas-api-xks-ca" pod "c26fc0e5-bd68-4d01-9fb8-9c16c1971b74" (UID: "c26fc0e5-bd68-4d01-9fb8-9c16c1971b74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:09.017585 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.017555 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c26fc0e5-bd68-4d01-9fb8-9c16c1971b74-kube-api-access-68djv" pod "c26fc0e5-bd68-4d01-9fb8-9c16c1971b74" (UID: "c26fc0e5-bd68-4d01-9fb8-9c16c1971b74"). InnerVolumeSpecName "kube-api-access-68djv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:09.117596 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.117558 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68djv\" (UniqueName: \"kubernetes.io/projected/c26fc0e5-bd68-4d01-9fb8-9c16c1971b74-kube-api-access-68djv\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:09.117596 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.117597 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26fc0e5-bd68-4d01-9fb8-9c16c1971b74-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:09.117850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.117612 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26fc0e5-bd68-4d01-9fb8-9c16c1971b74-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:09.213650 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.213614 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06e3ae2e-pp6k5"] Oct 30 21:39:09.214117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.214104 5801 cpu_manager.go:414] "containerMap: removing container" podUID="c26fc0e5-bd68-4d01-9fb8-9c16c1971b74" containerName="revoke-keys" Oct 30 21:39:09.214164 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.214118 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c26fc0e5-bd68-4d01-9fb8-9c16c1971b74" containerName="revoke-keys" Oct 30 21:39:09.214226 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.214217 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="c26fc0e5-bd68-4d01-9fb8-9c16c1971b74" containerName="revoke-keys" Oct 30 21:39:09.218690 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.218668 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e3ae2e-pp6k5" Oct 30 21:39:09.230989 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.230961 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06e3ae2e-pp6k5"] Oct 30 21:39:09.256422 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.256385 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:09.296710 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.296647 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e44bde8-l5x9w"] Oct 30 21:39:09.320666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.320632 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d07a906-016e-49df-88a4-3d3e3bf2dd32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06e3ae2e-pp6k5\" (UID: \"5d07a906-016e-49df-88a4-3d3e3bf2dd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e3ae2e-pp6k5" Oct 30 21:39:09.320791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.320691 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wl72\" (UniqueName: \"kubernetes.io/projected/5d07a906-016e-49df-88a4-3d3e3bf2dd32-kube-api-access-6wl72\") pod \"maas-api-revoke-sub-06e3ae2e-pp6k5\" (UID: \"5d07a906-016e-49df-88a4-3d3e3bf2dd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e3ae2e-pp6k5" Oct 30 21:39:09.321002 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.320982 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d07a906-016e-49df-88a4-3d3e3bf2dd32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06e3ae2e-pp6k5\" (UID: \"5d07a906-016e-49df-88a4-3d3e3bf2dd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e3ae2e-pp6k5" Oct 30 21:39:09.422363 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.422325 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d07a906-016e-49df-88a4-3d3e3bf2dd32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06e3ae2e-pp6k5\" (UID: \"5d07a906-016e-49df-88a4-3d3e3bf2dd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e3ae2e-pp6k5" Oct 30 21:39:09.422536 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.422484 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d07a906-016e-49df-88a4-3d3e3bf2dd32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06e3ae2e-pp6k5\" (UID: \"5d07a906-016e-49df-88a4-3d3e3bf2dd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e3ae2e-pp6k5" Oct 30 21:39:09.422536 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.422520 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wl72\" (UniqueName: \"kubernetes.io/projected/5d07a906-016e-49df-88a4-3d3e3bf2dd32-kube-api-access-6wl72\") pod \"maas-api-revoke-sub-06e3ae2e-pp6k5\" (UID: \"5d07a906-016e-49df-88a4-3d3e3bf2dd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e3ae2e-pp6k5" Oct 30 21:39:09.423108 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.423090 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d07a906-016e-49df-88a4-3d3e3bf2dd32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06e3ae2e-pp6k5\" (UID: \"5d07a906-016e-49df-88a4-3d3e3bf2dd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e3ae2e-pp6k5" Oct 30 21:39:09.424861 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.424840 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d07a906-016e-49df-88a4-3d3e3bf2dd32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06e3ae2e-pp6k5\" (UID: \"5d07a906-016e-49df-88a4-3d3e3bf2dd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e3ae2e-pp6k5" Oct 30 21:39:09.435428 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.435363 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wl72\" (UniqueName: \"kubernetes.io/projected/5d07a906-016e-49df-88a4-3d3e3bf2dd32-kube-api-access-6wl72\") pod \"maas-api-revoke-sub-06e3ae2e-pp6k5\" (UID: \"5d07a906-016e-49df-88a4-3d3e3bf2dd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e3ae2e-pp6k5" Oct 30 21:39:09.532472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.532444 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e3ae2e-pp6k5" Oct 30 21:39:09.719182 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.719100 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b2286e6-5v87x" Oct 30 21:39:09.719182 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.719096 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b2286e6-5v87x" event={"ID":"c26fc0e5-bd68-4d01-9fb8-9c16c1971b74","Type":"ContainerDied","Data":"ca72c69df54ed94d82feff9de102f6b1e4d3739f8986558306c8cf9d319c2652"} Oct 30 21:39:09.719362 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.719226 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca72c69df54ed94d82feff9de102f6b1e4d3739f8986558306c8cf9d319c2652" Oct 30 21:39:09.720916 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.720887 5801 generic.go:356] "Generic (PLEG): container finished" podID="d067f135-fcc0-4578-bb69-6fb5a6ecf49e" containerID="ab80284f5438f212fa8b5a94e1c7e0fe4e7264e6dadaeb079d88e3c91a6a8241" exitCode=0 Oct 30 21:39:09.721033 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.720917 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8991907d-wxl9z" event={"ID":"d067f135-fcc0-4578-bb69-6fb5a6ecf49e","Type":"ContainerDied","Data":"ab80284f5438f212fa8b5a94e1c7e0fe4e7264e6dadaeb079d88e3c91a6a8241"} Oct 30 21:39:09.722777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.722738 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44bde8-l5x9w" event={"ID":"44c9fa99-b9a5-452c-b352-481a0cf4ec4a","Type":"ContainerStarted","Data":"00c6d3d478d1a8a93d31113546340998c9854a9582b338de78d071697b0d1418"} Oct 30 21:39:09.722893 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.722793 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44bde8-l5x9w" event={"ID":"44c9fa99-b9a5-452c-b352-481a0cf4ec4a","Type":"ContainerStarted","Data":"4d7bba3bde279d01ff18982a6b76b7b56622f3503771d77c320bbe426d280964"} Oct 30 21:39:09.757529 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.757472 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44bde8-l5x9w" podStartSLOduration=1.757458632 podStartE2EDuration="1.757458632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:09.756811721 +0000 UTC m=+618.352628398" watchObservedRunningTime="2026-10-30 21:39:09.757458632 +0000 UTC m=+618.353275354" Oct 30 21:39:09.815250 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.815219 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06e3ae2e-pp6k5"] Oct 30 21:39:09.815362 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:39:09.815270 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d07a906_016e_49df_88a4_3d3e3bf2dd32.slice/crio-780c51beed2a49816bfb113600db6508c1dff09c075806296c0204ef43166e88 WatchSource:0}: Error finding container 780c51beed2a49816bfb113600db6508c1dff09c075806296c0204ef43166e88: Status 404 returned error can't find the container with id 780c51beed2a49816bfb113600db6508c1dff09c075806296c0204ef43166e88 Oct 30 21:39:09.859004 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.858981 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7d1d1-9xnnw" Oct 30 21:39:09.972360 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:09.972279 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:10.030668 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.030634 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f-maas-api-openshift-ca\") pod \"f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f\" (UID: \"f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f\") " Oct 30 21:39:10.031184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.030682 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f-maas-api-xks-ca\") pod \"f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f\" (UID: \"f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f\") " Oct 30 21:39:10.031184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.030826 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f-kube-api-access-jxhw5\" (UniqueName: \"kubernetes.io/projected/f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f-kube-api-access-jxhw5\") pod \"f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f\" (UID: \"f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f\") " Oct 30 21:39:10.031184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.031110 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f-maas-api-openshift-ca" pod "f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f" (UID: "f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:10.031380 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.031330 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:10.053474 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.053439 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f-maas-api-xks-ca" pod "f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f" (UID: "f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:10.053474 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.053439 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f-kube-api-access-jxhw5" pod "f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f" (UID: "f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f"). InnerVolumeSpecName "kube-api-access-jxhw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:10.132920 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.132888 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:10.132920 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.132916 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxhw5\" (UniqueName: \"kubernetes.io/projected/f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f-kube-api-access-jxhw5\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:10.730791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.730742 5801 generic.go:356] "Generic (PLEG): container finished" podID="44c9fa99-b9a5-452c-b352-481a0cf4ec4a" containerID="00c6d3d478d1a8a93d31113546340998c9854a9582b338de78d071697b0d1418" exitCode=0 Oct 30 21:39:10.730942 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.730798 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44bde8-l5x9w" event={"ID":"44c9fa99-b9a5-452c-b352-481a0cf4ec4a","Type":"ContainerDied","Data":"00c6d3d478d1a8a93d31113546340998c9854a9582b338de78d071697b0d1418"} Oct 30 21:39:10.732374 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.732351 5801 generic.go:356] "Generic (PLEG): container finished" podID="5d07a906-016e-49df-88a4-3d3e3bf2dd32" containerID="659824c8255b925dd92d9f7974d2532ca8d8c3a6eb918b36626c347ede86fd75" exitCode=0 Oct 30 21:39:10.732472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.732429 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e3ae2e-pp6k5" event={"ID":"5d07a906-016e-49df-88a4-3d3e3bf2dd32","Type":"ContainerDied","Data":"659824c8255b925dd92d9f7974d2532ca8d8c3a6eb918b36626c347ede86fd75"} Oct 30 21:39:10.732536 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.732471 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e3ae2e-pp6k5" event={"ID":"5d07a906-016e-49df-88a4-3d3e3bf2dd32","Type":"ContainerStarted","Data":"780c51beed2a49816bfb113600db6508c1dff09c075806296c0204ef43166e88"} Oct 30 21:39:10.733714 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.733698 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7d1d1-9xnnw" Oct 30 21:39:10.733804 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.733733 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7d1d1-9xnnw" event={"ID":"f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f","Type":"ContainerDied","Data":"ddcf39eda13cba9a19a68b08e491b0476fb0cea2df59c3f5cd2a5fea6efaa03c"} Oct 30 21:39:10.733804 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.733787 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddcf39eda13cba9a19a68b08e491b0476fb0cea2df59c3f5cd2a5fea6efaa03c" Oct 30 21:39:10.751940 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.751911 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:10.895629 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.895599 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ae7f502-ppw8w"] Oct 30 21:39:10.896377 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.896359 5801 cpu_manager.go:414] "containerMap: removing container" podUID="f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f" containerName="revoke-keys" Oct 30 21:39:10.896420 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.896380 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f" containerName="revoke-keys" Oct 30 21:39:10.896533 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.896522 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f" containerName="revoke-keys" Oct 30 21:39:10.900413 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.900391 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae7f502-ppw8w" Oct 30 21:39:10.908987 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.908958 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ae7f502-ppw8w"] Oct 30 21:39:10.922191 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:10.922164 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8991907d-wxl9z" Oct 30 21:39:11.042179 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.042138 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d067f135-fcc0-4578-bb69-6fb5a6ecf49e-kube-api-access-d8d7q\" (UniqueName: \"kubernetes.io/projected/d067f135-fcc0-4578-bb69-6fb5a6ecf49e-kube-api-access-d8d7q\") pod \"d067f135-fcc0-4578-bb69-6fb5a6ecf49e\" (UID: \"d067f135-fcc0-4578-bb69-6fb5a6ecf49e\") " Oct 30 21:39:11.042645 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.042313 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d067f135-fcc0-4578-bb69-6fb5a6ecf49e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d067f135-fcc0-4578-bb69-6fb5a6ecf49e-maas-api-xks-ca\") pod \"d067f135-fcc0-4578-bb69-6fb5a6ecf49e\" (UID: \"d067f135-fcc0-4578-bb69-6fb5a6ecf49e\") " Oct 30 21:39:11.042645 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.042378 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d067f135-fcc0-4578-bb69-6fb5a6ecf49e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d067f135-fcc0-4578-bb69-6fb5a6ecf49e-maas-api-openshift-ca\") pod \"d067f135-fcc0-4578-bb69-6fb5a6ecf49e\" (UID: \"d067f135-fcc0-4578-bb69-6fb5a6ecf49e\") " Oct 30 21:39:11.042812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.042649 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2191dc8-cb0d-4967-a182-32d50b9ad13e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ae7f502-ppw8w\" (UID: \"b2191dc8-cb0d-4967-a182-32d50b9ad13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae7f502-ppw8w" Oct 30 21:39:11.042812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.042682 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2191dc8-cb0d-4967-a182-32d50b9ad13e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ae7f502-ppw8w\" (UID: \"b2191dc8-cb0d-4967-a182-32d50b9ad13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae7f502-ppw8w" Oct 30 21:39:11.042933 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.042822 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gkxz\" (UniqueName: \"kubernetes.io/projected/b2191dc8-cb0d-4967-a182-32d50b9ad13e-kube-api-access-4gkxz\") pod \"maas-api-revoke-sub-3ae7f502-ppw8w\" (UID: \"b2191dc8-cb0d-4967-a182-32d50b9ad13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae7f502-ppw8w" Oct 30 21:39:11.042933 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.042886 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d067f135-fcc0-4578-bb69-6fb5a6ecf49e-maas-api-openshift-ca" pod "d067f135-fcc0-4578-bb69-6fb5a6ecf49e" (UID: "d067f135-fcc0-4578-bb69-6fb5a6ecf49e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:11.044588 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.044565 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d067f135-fcc0-4578-bb69-6fb5a6ecf49e-kube-api-access-d8d7q" pod "d067f135-fcc0-4578-bb69-6fb5a6ecf49e" (UID: "d067f135-fcc0-4578-bb69-6fb5a6ecf49e"). InnerVolumeSpecName "kube-api-access-d8d7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:11.045048 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.045027 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d067f135-fcc0-4578-bb69-6fb5a6ecf49e-maas-api-xks-ca" pod "d067f135-fcc0-4578-bb69-6fb5a6ecf49e" (UID: "d067f135-fcc0-4578-bb69-6fb5a6ecf49e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:11.144498 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.144458 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2191dc8-cb0d-4967-a182-32d50b9ad13e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ae7f502-ppw8w\" (UID: \"b2191dc8-cb0d-4967-a182-32d50b9ad13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae7f502-ppw8w" Oct 30 21:39:11.144678 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.144561 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gkxz\" (UniqueName: \"kubernetes.io/projected/b2191dc8-cb0d-4967-a182-32d50b9ad13e-kube-api-access-4gkxz\") pod \"maas-api-revoke-sub-3ae7f502-ppw8w\" (UID: \"b2191dc8-cb0d-4967-a182-32d50b9ad13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae7f502-ppw8w" Oct 30 21:39:11.144717 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.144694 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2191dc8-cb0d-4967-a182-32d50b9ad13e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ae7f502-ppw8w\" (UID: \"b2191dc8-cb0d-4967-a182-32d50b9ad13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae7f502-ppw8w" Oct 30 21:39:11.144783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.144734 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d067f135-fcc0-4578-bb69-6fb5a6ecf49e-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:11.144783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.144763 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d067f135-fcc0-4578-bb69-6fb5a6ecf49e-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:11.144848 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.144780 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8d7q\" (UniqueName: \"kubernetes.io/projected/d067f135-fcc0-4578-bb69-6fb5a6ecf49e-kube-api-access-d8d7q\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:11.145245 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.145225 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2191dc8-cb0d-4967-a182-32d50b9ad13e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ae7f502-ppw8w\" (UID: \"b2191dc8-cb0d-4967-a182-32d50b9ad13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae7f502-ppw8w" Oct 30 21:39:11.147034 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.147017 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2191dc8-cb0d-4967-a182-32d50b9ad13e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ae7f502-ppw8w\" (UID: \"b2191dc8-cb0d-4967-a182-32d50b9ad13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae7f502-ppw8w" Oct 30 21:39:11.156921 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.156899 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gkxz\" (UniqueName: \"kubernetes.io/projected/b2191dc8-cb0d-4967-a182-32d50b9ad13e-kube-api-access-4gkxz\") pod \"maas-api-revoke-sub-3ae7f502-ppw8w\" (UID: \"b2191dc8-cb0d-4967-a182-32d50b9ad13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae7f502-ppw8w" Oct 30 21:39:11.233868 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.233833 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae7f502-ppw8w" Oct 30 21:39:11.513429 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.513398 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ae7f502-ppw8w"] Oct 30 21:39:11.514615 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:39:11.514580 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2191dc8_cb0d_4967_a182_32d50b9ad13e.slice/crio-4494b6c259488078d8c96aa4fadee61af744a20d590d0a8bb8cd8bc1c9c82db4 WatchSource:0}: Error finding container 4494b6c259488078d8c96aa4fadee61af744a20d590d0a8bb8cd8bc1c9c82db4: Status 404 returned error can't find the container with id 4494b6c259488078d8c96aa4fadee61af744a20d590d0a8bb8cd8bc1c9c82db4 Oct 30 21:39:11.581967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.581934 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:11.741270 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.741233 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae7f502-ppw8w" event={"ID":"b2191dc8-cb0d-4967-a182-32d50b9ad13e","Type":"ContainerStarted","Data":"4494b6c259488078d8c96aa4fadee61af744a20d590d0a8bb8cd8bc1c9c82db4"} Oct 30 21:39:11.742507 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.742471 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8991907d-wxl9z" event={"ID":"d067f135-fcc0-4578-bb69-6fb5a6ecf49e","Type":"ContainerDied","Data":"1a70a8ad2017b7e65d473fafb928fdc90810183968930fda2d256d48b5c98a39"} Oct 30 21:39:11.742624 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.742520 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a70a8ad2017b7e65d473fafb928fdc90810183968930fda2d256d48b5c98a39" Oct 30 21:39:11.743020 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.742704 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8991907d-wxl9z" Oct 30 21:39:11.874967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.874940 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e3ae2e-pp6k5" Oct 30 21:39:11.920914 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:11.920888 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44bde8-l5x9w" Oct 30 21:39:12.056256 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.056225 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d07a906-016e-49df-88a4-3d3e3bf2dd32-kube-api-access-6wl72\" (UniqueName: \"kubernetes.io/projected/5d07a906-016e-49df-88a4-3d3e3bf2dd32-kube-api-access-6wl72\") pod \"5d07a906-016e-49df-88a4-3d3e3bf2dd32\" (UID: \"5d07a906-016e-49df-88a4-3d3e3bf2dd32\") " Oct 30 21:39:12.056692 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.056288 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44c9fa99-b9a5-452c-b352-481a0cf4ec4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c9fa99-b9a5-452c-b352-481a0cf4ec4a-maas-api-openshift-ca\") pod \"44c9fa99-b9a5-452c-b352-481a0cf4ec4a\" (UID: \"44c9fa99-b9a5-452c-b352-481a0cf4ec4a\") " Oct 30 21:39:12.056692 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.056316 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c9fa99-b9a5-452c-b352-481a0cf4ec4a-kube-api-access-wbjkr\" (UniqueName: \"kubernetes.io/projected/44c9fa99-b9a5-452c-b352-481a0cf4ec4a-kube-api-access-wbjkr\") pod \"44c9fa99-b9a5-452c-b352-481a0cf4ec4a\" (UID: \"44c9fa99-b9a5-452c-b352-481a0cf4ec4a\") " Oct 30 21:39:12.056692 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.056416 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d07a906-016e-49df-88a4-3d3e3bf2dd32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d07a906-016e-49df-88a4-3d3e3bf2dd32-maas-api-xks-ca\") pod \"5d07a906-016e-49df-88a4-3d3e3bf2dd32\" (UID: \"5d07a906-016e-49df-88a4-3d3e3bf2dd32\") " Oct 30 21:39:12.056692 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.056443 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d07a906-016e-49df-88a4-3d3e3bf2dd32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d07a906-016e-49df-88a4-3d3e3bf2dd32-maas-api-openshift-ca\") pod \"5d07a906-016e-49df-88a4-3d3e3bf2dd32\" (UID: \"5d07a906-016e-49df-88a4-3d3e3bf2dd32\") " Oct 30 21:39:12.056692 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.056574 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44c9fa99-b9a5-452c-b352-481a0cf4ec4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c9fa99-b9a5-452c-b352-481a0cf4ec4a-maas-api-xks-ca\") pod \"44c9fa99-b9a5-452c-b352-481a0cf4ec4a\" (UID: \"44c9fa99-b9a5-452c-b352-481a0cf4ec4a\") " Oct 30 21:39:12.056692 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.056675 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44c9fa99-b9a5-452c-b352-481a0cf4ec4a-maas-api-openshift-ca" pod "44c9fa99-b9a5-452c-b352-481a0cf4ec4a" (UID: "44c9fa99-b9a5-452c-b352-481a0cf4ec4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:12.057070 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.056869 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d07a906-016e-49df-88a4-3d3e3bf2dd32-maas-api-openshift-ca" pod "5d07a906-016e-49df-88a4-3d3e3bf2dd32" (UID: "5d07a906-016e-49df-88a4-3d3e3bf2dd32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:12.057132 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.057097 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c9fa99-b9a5-452c-b352-481a0cf4ec4a-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:12.057132 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.057120 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d07a906-016e-49df-88a4-3d3e3bf2dd32-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:12.058579 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.058542 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c9fa99-b9a5-452c-b352-481a0cf4ec4a-kube-api-access-wbjkr" pod "44c9fa99-b9a5-452c-b352-481a0cf4ec4a" (UID: "44c9fa99-b9a5-452c-b352-481a0cf4ec4a"). InnerVolumeSpecName "kube-api-access-wbjkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:12.058579 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.058567 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d07a906-016e-49df-88a4-3d3e3bf2dd32-kube-api-access-6wl72" pod "5d07a906-016e-49df-88a4-3d3e3bf2dd32" (UID: "5d07a906-016e-49df-88a4-3d3e3bf2dd32"). InnerVolumeSpecName "kube-api-access-6wl72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:12.058714 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.058601 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d07a906-016e-49df-88a4-3d3e3bf2dd32-maas-api-xks-ca" pod "5d07a906-016e-49df-88a4-3d3e3bf2dd32" (UID: "5d07a906-016e-49df-88a4-3d3e3bf2dd32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:12.058714 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.058603 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44c9fa99-b9a5-452c-b352-481a0cf4ec4a-maas-api-xks-ca" pod "44c9fa99-b9a5-452c-b352-481a0cf4ec4a" (UID: "44c9fa99-b9a5-452c-b352-481a0cf4ec4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:12.157846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.157811 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c9fa99-b9a5-452c-b352-481a0cf4ec4a-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:12.157846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.157843 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wl72\" (UniqueName: \"kubernetes.io/projected/5d07a906-016e-49df-88a4-3d3e3bf2dd32-kube-api-access-6wl72\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:12.158041 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.157858 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbjkr\" (UniqueName: \"kubernetes.io/projected/44c9fa99-b9a5-452c-b352-481a0cf4ec4a-kube-api-access-wbjkr\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:12.158041 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.157873 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d07a906-016e-49df-88a4-3d3e3bf2dd32-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:12.336090 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.335998 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:12.750816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.750733 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44bde8-l5x9w" Oct 30 21:39:12.750959 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.750733 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44bde8-l5x9w" event={"ID":"44c9fa99-b9a5-452c-b352-481a0cf4ec4a","Type":"ContainerDied","Data":"4d7bba3bde279d01ff18982a6b76b7b56622f3503771d77c320bbe426d280964"} Oct 30 21:39:12.750959 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.750869 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d7bba3bde279d01ff18982a6b76b7b56622f3503771d77c320bbe426d280964" Oct 30 21:39:12.752183 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.752163 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e3ae2e-pp6k5" Oct 30 21:39:12.752295 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.752203 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e3ae2e-pp6k5" event={"ID":"5d07a906-016e-49df-88a4-3d3e3bf2dd32","Type":"ContainerDied","Data":"780c51beed2a49816bfb113600db6508c1dff09c075806296c0204ef43166e88"} Oct 30 21:39:12.752295 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.752248 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="780c51beed2a49816bfb113600db6508c1dff09c075806296c0204ef43166e88" Oct 30 21:39:12.753924 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.753901 5801 generic.go:356] "Generic (PLEG): container finished" podID="b2191dc8-cb0d-4967-a182-32d50b9ad13e" containerID="022207ac1bf43d748cfba53ba9a53be8cbb35e9fb042611963bba2819e94c7cd" exitCode=0 Oct 30 21:39:12.754005 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:12.753989 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae7f502-ppw8w" event={"ID":"b2191dc8-cb0d-4967-a182-32d50b9ad13e","Type":"ContainerDied","Data":"022207ac1bf43d748cfba53ba9a53be8cbb35e9fb042611963bba2819e94c7cd"} Oct 30 21:39:13.149084 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:13.149025 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:13.908467 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:13.908442 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae7f502-ppw8w" Oct 30 21:39:13.974444 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:13.974404 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2191dc8-cb0d-4967-a182-32d50b9ad13e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2191dc8-cb0d-4967-a182-32d50b9ad13e-maas-api-openshift-ca\") pod \"b2191dc8-cb0d-4967-a182-32d50b9ad13e\" (UID: \"b2191dc8-cb0d-4967-a182-32d50b9ad13e\") " Oct 30 21:39:13.974632 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:13.974493 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2191dc8-cb0d-4967-a182-32d50b9ad13e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2191dc8-cb0d-4967-a182-32d50b9ad13e-maas-api-xks-ca\") pod \"b2191dc8-cb0d-4967-a182-32d50b9ad13e\" (UID: \"b2191dc8-cb0d-4967-a182-32d50b9ad13e\") " Oct 30 21:39:13.974728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:13.974655 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2191dc8-cb0d-4967-a182-32d50b9ad13e-kube-api-access-4gkxz\" (UniqueName: \"kubernetes.io/projected/b2191dc8-cb0d-4967-a182-32d50b9ad13e-kube-api-access-4gkxz\") pod \"b2191dc8-cb0d-4967-a182-32d50b9ad13e\" (UID: \"b2191dc8-cb0d-4967-a182-32d50b9ad13e\") " Oct 30 21:39:13.974816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:13.974739 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2191dc8-cb0d-4967-a182-32d50b9ad13e-maas-api-openshift-ca" pod "b2191dc8-cb0d-4967-a182-32d50b9ad13e" (UID: "b2191dc8-cb0d-4967-a182-32d50b9ad13e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:13.975112 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:13.975092 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2191dc8-cb0d-4967-a182-32d50b9ad13e-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:13.976644 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:13.976618 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2191dc8-cb0d-4967-a182-32d50b9ad13e-maas-api-xks-ca" pod "b2191dc8-cb0d-4967-a182-32d50b9ad13e" (UID: "b2191dc8-cb0d-4967-a182-32d50b9ad13e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:13.976960 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:13.976942 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2191dc8-cb0d-4967-a182-32d50b9ad13e-kube-api-access-4gkxz" pod "b2191dc8-cb0d-4967-a182-32d50b9ad13e" (UID: "b2191dc8-cb0d-4967-a182-32d50b9ad13e"). InnerVolumeSpecName "kube-api-access-4gkxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:14.076403 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:14.076364 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2191dc8-cb0d-4967-a182-32d50b9ad13e-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:14.076403 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:14.076398 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gkxz\" (UniqueName: \"kubernetes.io/projected/b2191dc8-cb0d-4967-a182-32d50b9ad13e-kube-api-access-4gkxz\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:14.094742 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:14.094704 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:14.767836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:14.767801 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae7f502-ppw8w" Oct 30 21:39:14.767836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:14.767821 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae7f502-ppw8w" event={"ID":"b2191dc8-cb0d-4967-a182-32d50b9ad13e","Type":"ContainerDied","Data":"4494b6c259488078d8c96aa4fadee61af744a20d590d0a8bb8cd8bc1c9c82db4"} Oct 30 21:39:14.768307 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:14.767873 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4494b6c259488078d8c96aa4fadee61af744a20d590d0a8bb8cd8bc1c9c82db4" Oct 30 21:39:14.964724 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:14.964618 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:16.126777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:16.125991 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:16.994776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:16.993852 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:17.410778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:17.410061 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:17.467886 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:17.463689 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:18.225797 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:18.225417 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:18.635817 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:18.634500 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:19.539515 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:19.539476 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:20.360764 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:20.360714 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:20.963582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:20.963548 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d80c67-945j6"] Oct 30 21:39:20.964229 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:20.964212 5801 cpu_manager.go:414] "containerMap: removing container" podUID="d067f135-fcc0-4578-bb69-6fb5a6ecf49e" containerName="revoke-keys" Oct 30 21:39:20.964306 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:20.964231 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d067f135-fcc0-4578-bb69-6fb5a6ecf49e" containerName="revoke-keys" Oct 30 21:39:20.964306 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:20.964257 5801 cpu_manager.go:414] "containerMap: removing container" podUID="44c9fa99-b9a5-452c-b352-481a0cf4ec4a" containerName="revoke-keys" Oct 30 21:39:20.964306 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:20.964265 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c9fa99-b9a5-452c-b352-481a0cf4ec4a" containerName="revoke-keys" Oct 30 21:39:20.964306 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:20.964292 5801 cpu_manager.go:414] "containerMap: removing container" podUID="b2191dc8-cb0d-4967-a182-32d50b9ad13e" containerName="revoke-keys" Oct 30 21:39:20.964306 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:20.964299 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2191dc8-cb0d-4967-a182-32d50b9ad13e" containerName="revoke-keys" Oct 30 21:39:20.964455 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:20.964311 5801 cpu_manager.go:414] "containerMap: removing container" podUID="5d07a906-016e-49df-88a4-3d3e3bf2dd32" containerName="revoke-keys" Oct 30 21:39:20.964455 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:20.964318 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d07a906-016e-49df-88a4-3d3e3bf2dd32" containerName="revoke-keys" Oct 30 21:39:20.964455 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:20.964425 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="d067f135-fcc0-4578-bb69-6fb5a6ecf49e" containerName="revoke-keys" Oct 30 21:39:20.964455 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:20.964440 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="44c9fa99-b9a5-452c-b352-481a0cf4ec4a" containerName="revoke-keys" Oct 30 21:39:20.964455 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:20.964453 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2191dc8-cb0d-4967-a182-32d50b9ad13e" containerName="revoke-keys" Oct 30 21:39:20.964586 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:20.964462 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d07a906-016e-49df-88a4-3d3e3bf2dd32" containerName="revoke-keys" Oct 30 21:39:20.972218 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:20.972194 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d80c67-945j6" Oct 30 21:39:20.978779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:20.978737 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d80c67-945j6"] Oct 30 21:39:20.981711 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:20.981686 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:39:20.981867 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:20.981726 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:39:20.981867 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:20.981727 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:39:20.982056 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:20.981974 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:39:21.044503 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.044467 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-685bdabe-hljnv"] Oct 30 21:39:21.048480 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.048460 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685bdabe-hljnv" Oct 30 21:39:21.057865 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.057841 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-685bdabe-hljnv"] Oct 30 21:39:21.068240 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.068215 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e18b72ce-2635-4754-9cfb-880e5970a807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-685bdabe-hljnv\" (UID: \"e18b72ce-2635-4754-9cfb-880e5970a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685bdabe-hljnv" Oct 30 21:39:21.068354 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.068244 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e18b72ce-2635-4754-9cfb-880e5970a807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-685bdabe-hljnv\" (UID: \"e18b72ce-2635-4754-9cfb-880e5970a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685bdabe-hljnv" Oct 30 21:39:21.068354 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.068289 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w2j7\" (UniqueName: \"kubernetes.io/projected/596fe353-a183-4cb7-9fe7-b4d9cecf90aa-kube-api-access-8w2j7\") pod \"maas-api-revoke-sub-01d80c67-945j6\" (UID: \"596fe353-a183-4cb7-9fe7-b4d9cecf90aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d80c67-945j6" Oct 30 21:39:21.068505 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.068479 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plhgt\" (UniqueName: \"kubernetes.io/projected/e18b72ce-2635-4754-9cfb-880e5970a807-kube-api-access-plhgt\") pod \"maas-api-revoke-sub-685bdabe-hljnv\" (UID: \"e18b72ce-2635-4754-9cfb-880e5970a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685bdabe-hljnv" Oct 30 21:39:21.068632 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.068614 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596fe353-a183-4cb7-9fe7-b4d9cecf90aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01d80c67-945j6\" (UID: \"596fe353-a183-4cb7-9fe7-b4d9cecf90aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d80c67-945j6" Oct 30 21:39:21.068706 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.068690 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596fe353-a183-4cb7-9fe7-b4d9cecf90aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01d80c67-945j6\" (UID: \"596fe353-a183-4cb7-9fe7-b4d9cecf90aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d80c67-945j6" Oct 30 21:39:21.169771 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.169725 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e18b72ce-2635-4754-9cfb-880e5970a807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-685bdabe-hljnv\" (UID: \"e18b72ce-2635-4754-9cfb-880e5970a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685bdabe-hljnv" Oct 30 21:39:21.169977 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.169792 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e18b72ce-2635-4754-9cfb-880e5970a807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-685bdabe-hljnv\" (UID: \"e18b72ce-2635-4754-9cfb-880e5970a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685bdabe-hljnv" Oct 30 21:39:21.169977 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.169867 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w2j7\" (UniqueName: \"kubernetes.io/projected/596fe353-a183-4cb7-9fe7-b4d9cecf90aa-kube-api-access-8w2j7\") pod \"maas-api-revoke-sub-01d80c67-945j6\" (UID: \"596fe353-a183-4cb7-9fe7-b4d9cecf90aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d80c67-945j6" Oct 30 21:39:21.169977 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.169915 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plhgt\" (UniqueName: \"kubernetes.io/projected/e18b72ce-2635-4754-9cfb-880e5970a807-kube-api-access-plhgt\") pod \"maas-api-revoke-sub-685bdabe-hljnv\" (UID: \"e18b72ce-2635-4754-9cfb-880e5970a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685bdabe-hljnv" Oct 30 21:39:21.170127 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.170020 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596fe353-a183-4cb7-9fe7-b4d9cecf90aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01d80c67-945j6\" (UID: \"596fe353-a183-4cb7-9fe7-b4d9cecf90aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d80c67-945j6" Oct 30 21:39:21.170127 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.170080 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596fe353-a183-4cb7-9fe7-b4d9cecf90aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01d80c67-945j6\" (UID: \"596fe353-a183-4cb7-9fe7-b4d9cecf90aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d80c67-945j6" Oct 30 21:39:21.170557 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.170525 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e18b72ce-2635-4754-9cfb-880e5970a807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-685bdabe-hljnv\" (UID: \"e18b72ce-2635-4754-9cfb-880e5970a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685bdabe-hljnv" Oct 30 21:39:21.170742 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.170713 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596fe353-a183-4cb7-9fe7-b4d9cecf90aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01d80c67-945j6\" (UID: \"596fe353-a183-4cb7-9fe7-b4d9cecf90aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d80c67-945j6" Oct 30 21:39:21.172369 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.172341 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e18b72ce-2635-4754-9cfb-880e5970a807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-685bdabe-hljnv\" (UID: \"e18b72ce-2635-4754-9cfb-880e5970a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685bdabe-hljnv" Oct 30 21:39:21.172466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.172384 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596fe353-a183-4cb7-9fe7-b4d9cecf90aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01d80c67-945j6\" (UID: \"596fe353-a183-4cb7-9fe7-b4d9cecf90aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d80c67-945j6" Oct 30 21:39:21.182556 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.182519 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w2j7\" (UniqueName: \"kubernetes.io/projected/596fe353-a183-4cb7-9fe7-b4d9cecf90aa-kube-api-access-8w2j7\") pod \"maas-api-revoke-sub-01d80c67-945j6\" (UID: \"596fe353-a183-4cb7-9fe7-b4d9cecf90aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d80c67-945j6" Oct 30 21:39:21.182556 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.182534 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plhgt\" (UniqueName: \"kubernetes.io/projected/e18b72ce-2635-4754-9cfb-880e5970a807-kube-api-access-plhgt\") pod \"maas-api-revoke-sub-685bdabe-hljnv\" (UID: \"e18b72ce-2635-4754-9cfb-880e5970a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685bdabe-hljnv" Oct 30 21:39:21.278907 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.278810 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67e59534-wh8w6"] Oct 30 21:39:21.286732 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.286703 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e59534-wh8w6" Oct 30 21:39:21.288602 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.288571 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67e59534-wh8w6"] Oct 30 21:39:21.288723 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.288686 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d80c67-945j6" Oct 30 21:39:21.361731 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.361697 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfad835c-c46dw"] Oct 30 21:39:21.365401 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.365020 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685bdabe-hljnv" Oct 30 21:39:21.375364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.374028 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfad835c-c46dw" Oct 30 21:39:21.379187 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.378844 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfad835c-c46dw"] Oct 30 21:39:21.379691 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.379669 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67e59534-wh8w6\" (UID: \"89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e59534-wh8w6" Oct 30 21:39:21.379850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.379779 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67e59534-wh8w6\" (UID: \"89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e59534-wh8w6" Oct 30 21:39:21.379850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.379834 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd5m5\" (UniqueName: \"kubernetes.io/projected/89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1-kube-api-access-cd5m5\") pod \"maas-api-revoke-sub-67e59534-wh8w6\" (UID: \"89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e59534-wh8w6" Oct 30 21:39:21.480919 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.480879 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67e59534-wh8w6\" (UID: \"89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e59534-wh8w6" Oct 30 21:39:21.481104 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.480935 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhbts\" (UniqueName: \"kubernetes.io/projected/c38d4c28-d335-44ba-991d-37cb69011ca2-kube-api-access-dhbts\") pod \"maas-api-revoke-sub-cfad835c-c46dw\" (UID: \"c38d4c28-d335-44ba-991d-37cb69011ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfad835c-c46dw" Oct 30 21:39:21.481104 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.480990 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c38d4c28-d335-44ba-991d-37cb69011ca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfad835c-c46dw\" (UID: \"c38d4c28-d335-44ba-991d-37cb69011ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfad835c-c46dw" Oct 30 21:39:21.481104 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.481045 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67e59534-wh8w6\" (UID: \"89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e59534-wh8w6" Oct 30 21:39:21.481104 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.481083 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c38d4c28-d335-44ba-991d-37cb69011ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfad835c-c46dw\" (UID: \"c38d4c28-d335-44ba-991d-37cb69011ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfad835c-c46dw" Oct 30 21:39:21.481328 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.481123 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd5m5\" (UniqueName: \"kubernetes.io/projected/89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1-kube-api-access-cd5m5\") pod \"maas-api-revoke-sub-67e59534-wh8w6\" (UID: \"89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e59534-wh8w6" Oct 30 21:39:21.481641 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.481608 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67e59534-wh8w6\" (UID: \"89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e59534-wh8w6" Oct 30 21:39:21.484143 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.484122 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67e59534-wh8w6\" (UID: \"89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e59534-wh8w6" Oct 30 21:39:21.498784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.498738 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd5m5\" (UniqueName: \"kubernetes.io/projected/89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1-kube-api-access-cd5m5\") pod \"maas-api-revoke-sub-67e59534-wh8w6\" (UID: \"89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e59534-wh8w6" Oct 30 21:39:21.541436 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.541359 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:21.577570 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:39:21.577540 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod596fe353_a183_4cb7_9fe7_b4d9cecf90aa.slice/crio-9d0277c1eb49548ca6a8ac1c330f5f8169f63e9224a96a2e3663893f758f6035 WatchSource:0}: Error finding container 9d0277c1eb49548ca6a8ac1c330f5f8169f63e9224a96a2e3663893f758f6035: Status 404 returned error can't find the container with id 9d0277c1eb49548ca6a8ac1c330f5f8169f63e9224a96a2e3663893f758f6035 Oct 30 21:39:21.580035 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.579774 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d80c67-945j6"] Oct 30 21:39:21.582266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.582202 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhbts\" (UniqueName: \"kubernetes.io/projected/c38d4c28-d335-44ba-991d-37cb69011ca2-kube-api-access-dhbts\") pod \"maas-api-revoke-sub-cfad835c-c46dw\" (UID: \"c38d4c28-d335-44ba-991d-37cb69011ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfad835c-c46dw" Oct 30 21:39:21.582341 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.582283 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c38d4c28-d335-44ba-991d-37cb69011ca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfad835c-c46dw\" (UID: \"c38d4c28-d335-44ba-991d-37cb69011ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfad835c-c46dw" Oct 30 21:39:21.582433 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.582404 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c38d4c28-d335-44ba-991d-37cb69011ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfad835c-c46dw\" (UID: \"c38d4c28-d335-44ba-991d-37cb69011ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfad835c-c46dw" Oct 30 21:39:21.584238 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.583182 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c38d4c28-d335-44ba-991d-37cb69011ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfad835c-c46dw\" (UID: \"c38d4c28-d335-44ba-991d-37cb69011ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfad835c-c46dw" Oct 30 21:39:21.586410 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.586285 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c38d4c28-d335-44ba-991d-37cb69011ca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfad835c-c46dw\" (UID: \"c38d4c28-d335-44ba-991d-37cb69011ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfad835c-c46dw" Oct 30 21:39:21.599399 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.599367 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhbts\" (UniqueName: \"kubernetes.io/projected/c38d4c28-d335-44ba-991d-37cb69011ca2-kube-api-access-dhbts\") pod \"maas-api-revoke-sub-cfad835c-c46dw\" (UID: \"c38d4c28-d335-44ba-991d-37cb69011ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfad835c-c46dw" Oct 30 21:39:21.604914 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.604891 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e59534-wh8w6" Oct 30 21:39:21.655854 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.655816 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-685bdabe-hljnv"] Oct 30 21:39:21.657318 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:39:21.657279 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode18b72ce_2635_4754_9cfb_880e5970a807.slice/crio-43440de8728a13d151431d9edf2302bf583448dbb7d65e47b09fa8ec31e78453 WatchSource:0}: Error finding container 43440de8728a13d151431d9edf2302bf583448dbb7d65e47b09fa8ec31e78453: Status 404 returned error can't find the container with id 43440de8728a13d151431d9edf2302bf583448dbb7d65e47b09fa8ec31e78453 Oct 30 21:39:21.711721 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.711694 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfad835c-c46dw" Oct 30 21:39:21.892711 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.892679 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67e59534-wh8w6"] Oct 30 21:39:21.893521 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:39:21.893493 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89fb6e5b_34c8_4ef3_a938_fd5f2393eaf1.slice/crio-73b45999e2db462deb620034285e99d235d776fe71b959d68a6fd654d15a34ca WatchSource:0}: Error finding container 73b45999e2db462deb620034285e99d235d776fe71b959d68a6fd654d15a34ca: Status 404 returned error can't find the container with id 73b45999e2db462deb620034285e99d235d776fe71b959d68a6fd654d15a34ca Oct 30 21:39:21.971161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.971134 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e59534-wh8w6" event={"ID":"89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1","Type":"ContainerStarted","Data":"73b45999e2db462deb620034285e99d235d776fe71b959d68a6fd654d15a34ca"} Oct 30 21:39:21.972356 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.972332 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685bdabe-hljnv" event={"ID":"e18b72ce-2635-4754-9cfb-880e5970a807","Type":"ContainerStarted","Data":"43440de8728a13d151431d9edf2302bf583448dbb7d65e47b09fa8ec31e78453"} Oct 30 21:39:21.974592 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.974572 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d80c67-945j6" event={"ID":"596fe353-a183-4cb7-9fe7-b4d9cecf90aa","Type":"ContainerStarted","Data":"9d0277c1eb49548ca6a8ac1c330f5f8169f63e9224a96a2e3663893f758f6035"} Oct 30 21:39:21.996837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.996814 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfad835c-c46dw"] Oct 30 21:39:21.997409 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:21.997367 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d80c67-945j6" podStartSLOduration=1.9973501219999998 podStartE2EDuration="1.997350122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:21.994469844 +0000 UTC m=+630.590286504" watchObservedRunningTime="2026-10-30 21:39:21.997350122 +0000 UTC m=+630.593166785" Oct 30 21:39:22.056260 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:39:22.056190 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc38d4c28_d335_44ba_991d_37cb69011ca2.slice/crio-15f1d2ff18977d48ccfe061cbc0714a90af7f866435ff36077b5fbeed7b145e8 WatchSource:0}: Error finding container 15f1d2ff18977d48ccfe061cbc0714a90af7f866435ff36077b5fbeed7b145e8: Status 404 returned error can't find the container with id 15f1d2ff18977d48ccfe061cbc0714a90af7f866435ff36077b5fbeed7b145e8 Oct 30 21:39:22.540706 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:22.540669 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:22.982884 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:22.982828 5801 generic.go:356] "Generic (PLEG): container finished" podID="596fe353-a183-4cb7-9fe7-b4d9cecf90aa" containerID="af013c85e9423c77a066159faf99dffaee20e0ff35b3e1a4fad37cc0a22f356f" exitCode=0 Oct 30 21:39:22.983030 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:22.982907 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d80c67-945j6" event={"ID":"596fe353-a183-4cb7-9fe7-b4d9cecf90aa","Type":"ContainerDied","Data":"af013c85e9423c77a066159faf99dffaee20e0ff35b3e1a4fad37cc0a22f356f"} Oct 30 21:39:22.984460 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:22.984438 5801 generic.go:356] "Generic (PLEG): container finished" podID="89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1" containerID="32ca5e686e231e1a3f923052a2e121a890028b21d5dc9c20db9f75a2db66af9b" exitCode=0 Oct 30 21:39:22.984562 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:22.984506 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e59534-wh8w6" event={"ID":"89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1","Type":"ContainerDied","Data":"32ca5e686e231e1a3f923052a2e121a890028b21d5dc9c20db9f75a2db66af9b"} Oct 30 21:39:22.985947 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:22.985929 5801 generic.go:356] "Generic (PLEG): container finished" podID="e18b72ce-2635-4754-9cfb-880e5970a807" containerID="be99c0dd97261916fb151bd94d27f8448ebb2d7bc996ae7b00f069f75c6c8fe6" exitCode=0 Oct 30 21:39:22.986039 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:22.986004 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685bdabe-hljnv" event={"ID":"e18b72ce-2635-4754-9cfb-880e5970a807","Type":"ContainerDied","Data":"be99c0dd97261916fb151bd94d27f8448ebb2d7bc996ae7b00f069f75c6c8fe6"} Oct 30 21:39:22.987557 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:22.987540 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfad835c-c46dw" event={"ID":"c38d4c28-d335-44ba-991d-37cb69011ca2","Type":"ContainerStarted","Data":"edad792c0da173d2d283ed802512797603dac64c33e8e5f2e0d5e07b869dfd1b"} Oct 30 21:39:22.987649 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:22.987576 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfad835c-c46dw" event={"ID":"c38d4c28-d335-44ba-991d-37cb69011ca2","Type":"ContainerStarted","Data":"15f1d2ff18977d48ccfe061cbc0714a90af7f866435ff36077b5fbeed7b145e8"} Oct 30 21:39:23.019917 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:23.019865 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfad835c-c46dw" podStartSLOduration=2.019848707 podStartE2EDuration="2.019848707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:23.019161004 +0000 UTC m=+631.614977700" watchObservedRunningTime="2026-10-30 21:39:23.019848707 +0000 UTC m=+631.615665364" Oct 30 21:39:23.342664 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:23.342628 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:23.995306 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:23.995267 5801 generic.go:356] "Generic (PLEG): container finished" podID="c38d4c28-d335-44ba-991d-37cb69011ca2" containerID="edad792c0da173d2d283ed802512797603dac64c33e8e5f2e0d5e07b869dfd1b" exitCode=0 Oct 30 21:39:23.995778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:23.995351 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfad835c-c46dw" event={"ID":"c38d4c28-d335-44ba-991d-37cb69011ca2","Type":"ContainerDied","Data":"edad792c0da173d2d283ed802512797603dac64c33e8e5f2e0d5e07b869dfd1b"} Oct 30 21:39:24.183435 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.183404 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685bdabe-hljnv" Oct 30 21:39:24.220098 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.220069 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e18b72ce-2635-4754-9cfb-880e5970a807-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e18b72ce-2635-4754-9cfb-880e5970a807-maas-api-xks-ca\") pod \"e18b72ce-2635-4754-9cfb-880e5970a807\" (UID: \"e18b72ce-2635-4754-9cfb-880e5970a807\") " Oct 30 21:39:24.220249 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.220110 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e18b72ce-2635-4754-9cfb-880e5970a807-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e18b72ce-2635-4754-9cfb-880e5970a807-maas-api-openshift-ca\") pod \"e18b72ce-2635-4754-9cfb-880e5970a807\" (UID: \"e18b72ce-2635-4754-9cfb-880e5970a807\") " Oct 30 21:39:24.220249 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.220241 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e18b72ce-2635-4754-9cfb-880e5970a807-kube-api-access-plhgt\" (UniqueName: \"kubernetes.io/projected/e18b72ce-2635-4754-9cfb-880e5970a807-kube-api-access-plhgt\") pod \"e18b72ce-2635-4754-9cfb-880e5970a807\" (UID: \"e18b72ce-2635-4754-9cfb-880e5970a807\") " Oct 30 21:39:24.220541 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.220502 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e18b72ce-2635-4754-9cfb-880e5970a807-maas-api-openshift-ca" pod "e18b72ce-2635-4754-9cfb-880e5970a807" (UID: "e18b72ce-2635-4754-9cfb-880e5970a807"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:24.220664 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.220644 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e18b72ce-2635-4754-9cfb-880e5970a807-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:24.220811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.220796 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e59534-wh8w6" Oct 30 21:39:24.222226 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.222203 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e18b72ce-2635-4754-9cfb-880e5970a807-maas-api-xks-ca" pod "e18b72ce-2635-4754-9cfb-880e5970a807" (UID: "e18b72ce-2635-4754-9cfb-880e5970a807"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:24.222643 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.222620 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e18b72ce-2635-4754-9cfb-880e5970a807-kube-api-access-plhgt" pod "e18b72ce-2635-4754-9cfb-880e5970a807" (UID: "e18b72ce-2635-4754-9cfb-880e5970a807"). InnerVolumeSpecName "kube-api-access-plhgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:24.232774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.232727 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d80c67-945j6" Oct 30 21:39:24.242505 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.242398 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:24.321905 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.321872 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/596fe353-a183-4cb7-9fe7-b4d9cecf90aa-kube-api-access-8w2j7\" (UniqueName: \"kubernetes.io/projected/596fe353-a183-4cb7-9fe7-b4d9cecf90aa-kube-api-access-8w2j7\") pod \"596fe353-a183-4cb7-9fe7-b4d9cecf90aa\" (UID: \"596fe353-a183-4cb7-9fe7-b4d9cecf90aa\") " Oct 30 21:39:24.321905 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.321917 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1-kube-api-access-cd5m5\" (UniqueName: \"kubernetes.io/projected/89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1-kube-api-access-cd5m5\") pod \"89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1\" (UID: \"89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1\") " Oct 30 21:39:24.322195 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.321993 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/596fe353-a183-4cb7-9fe7-b4d9cecf90aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596fe353-a183-4cb7-9fe7-b4d9cecf90aa-maas-api-xks-ca\") pod \"596fe353-a183-4cb7-9fe7-b4d9cecf90aa\" (UID: \"596fe353-a183-4cb7-9fe7-b4d9cecf90aa\") " Oct 30 21:39:24.322195 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.322068 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1-maas-api-xks-ca\") pod \"89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1\" (UID: \"89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1\") " Oct 30 21:39:24.322195 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.322122 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1-maas-api-openshift-ca\") pod \"89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1\" (UID: \"89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1\") " Oct 30 21:39:24.322195 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.322184 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/596fe353-a183-4cb7-9fe7-b4d9cecf90aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596fe353-a183-4cb7-9fe7-b4d9cecf90aa-maas-api-openshift-ca\") pod \"596fe353-a183-4cb7-9fe7-b4d9cecf90aa\" (UID: \"596fe353-a183-4cb7-9fe7-b4d9cecf90aa\") " Oct 30 21:39:24.322599 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.322560 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1-maas-api-openshift-ca" pod "89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1" (UID: "89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:24.322737 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.322600 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plhgt\" (UniqueName: \"kubernetes.io/projected/e18b72ce-2635-4754-9cfb-880e5970a807-kube-api-access-plhgt\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:24.322737 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.322621 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e18b72ce-2635-4754-9cfb-880e5970a807-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:24.322737 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.322636 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/596fe353-a183-4cb7-9fe7-b4d9cecf90aa-maas-api-openshift-ca" pod "596fe353-a183-4cb7-9fe7-b4d9cecf90aa" (UID: "596fe353-a183-4cb7-9fe7-b4d9cecf90aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:24.324383 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.324351 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/596fe353-a183-4cb7-9fe7-b4d9cecf90aa-maas-api-xks-ca" pod "596fe353-a183-4cb7-9fe7-b4d9cecf90aa" (UID: "596fe353-a183-4cb7-9fe7-b4d9cecf90aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:24.324493 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.324440 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596fe353-a183-4cb7-9fe7-b4d9cecf90aa-kube-api-access-8w2j7" pod "596fe353-a183-4cb7-9fe7-b4d9cecf90aa" (UID: "596fe353-a183-4cb7-9fe7-b4d9cecf90aa"). InnerVolumeSpecName "kube-api-access-8w2j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:24.324493 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.324478 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1-kube-api-access-cd5m5" pod "89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1" (UID: "89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1"). InnerVolumeSpecName "kube-api-access-cd5m5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:24.324570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.324498 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1-maas-api-xks-ca" pod "89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1" (UID: "89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:24.423397 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.423356 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:24.423397 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.423387 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:24.423397 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.423399 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596fe353-a183-4cb7-9fe7-b4d9cecf90aa-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:24.423397 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.423408 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w2j7\" (UniqueName: \"kubernetes.io/projected/596fe353-a183-4cb7-9fe7-b4d9cecf90aa-kube-api-access-8w2j7\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:24.423667 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.423418 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd5m5\" (UniqueName: \"kubernetes.io/projected/89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1-kube-api-access-cd5m5\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:24.423667 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:24.423426 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596fe353-a183-4cb7-9fe7-b4d9cecf90aa-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:25.003528 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:25.003499 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d80c67-945j6" Oct 30 21:39:25.003528 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:25.003518 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d80c67-945j6" event={"ID":"596fe353-a183-4cb7-9fe7-b4d9cecf90aa","Type":"ContainerDied","Data":"9d0277c1eb49548ca6a8ac1c330f5f8169f63e9224a96a2e3663893f758f6035"} Oct 30 21:39:25.004060 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:25.003566 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d0277c1eb49548ca6a8ac1c330f5f8169f63e9224a96a2e3663893f758f6035" Oct 30 21:39:25.004916 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:25.004899 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e59534-wh8w6" Oct 30 21:39:25.004916 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:25.004911 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67e59534-wh8w6" event={"ID":"89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1","Type":"ContainerDied","Data":"73b45999e2db462deb620034285e99d235d776fe71b959d68a6fd654d15a34ca"} Oct 30 21:39:25.005070 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:25.004948 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73b45999e2db462deb620034285e99d235d776fe71b959d68a6fd654d15a34ca" Oct 30 21:39:25.006698 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:25.006677 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685bdabe-hljnv" Oct 30 21:39:25.006854 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:25.006829 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685bdabe-hljnv" event={"ID":"e18b72ce-2635-4754-9cfb-880e5970a807","Type":"ContainerDied","Data":"43440de8728a13d151431d9edf2302bf583448dbb7d65e47b09fa8ec31e78453"} Oct 30 21:39:25.006989 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:25.006875 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43440de8728a13d151431d9edf2302bf583448dbb7d65e47b09fa8ec31e78453" Oct 30 21:39:25.060163 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:25.060128 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:25.169146 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:25.169122 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfad835c-c46dw" Oct 30 21:39:25.232229 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:25.232190 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c38d4c28-d335-44ba-991d-37cb69011ca2-kube-api-access-dhbts\" (UniqueName: \"kubernetes.io/projected/c38d4c28-d335-44ba-991d-37cb69011ca2-kube-api-access-dhbts\") pod \"c38d4c28-d335-44ba-991d-37cb69011ca2\" (UID: \"c38d4c28-d335-44ba-991d-37cb69011ca2\") " Oct 30 21:39:25.232420 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:25.232275 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c38d4c28-d335-44ba-991d-37cb69011ca2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c38d4c28-d335-44ba-991d-37cb69011ca2-maas-api-xks-ca\") pod \"c38d4c28-d335-44ba-991d-37cb69011ca2\" (UID: \"c38d4c28-d335-44ba-991d-37cb69011ca2\") " Oct 30 21:39:25.232420 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:25.232403 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c38d4c28-d335-44ba-991d-37cb69011ca2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c38d4c28-d335-44ba-991d-37cb69011ca2-maas-api-openshift-ca\") pod \"c38d4c28-d335-44ba-991d-37cb69011ca2\" (UID: \"c38d4c28-d335-44ba-991d-37cb69011ca2\") " Oct 30 21:39:25.232839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:25.232808 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38d4c28-d335-44ba-991d-37cb69011ca2-maas-api-openshift-ca" pod "c38d4c28-d335-44ba-991d-37cb69011ca2" (UID: "c38d4c28-d335-44ba-991d-37cb69011ca2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:25.233030 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:25.233011 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c38d4c28-d335-44ba-991d-37cb69011ca2-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:25.234416 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:25.234385 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c38d4c28-d335-44ba-991d-37cb69011ca2-kube-api-access-dhbts" pod "c38d4c28-d335-44ba-991d-37cb69011ca2" (UID: "c38d4c28-d335-44ba-991d-37cb69011ca2"). InnerVolumeSpecName "kube-api-access-dhbts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:25.234584 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:25.234557 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c38d4c28-d335-44ba-991d-37cb69011ca2-maas-api-xks-ca" pod "c38d4c28-d335-44ba-991d-37cb69011ca2" (UID: "c38d4c28-d335-44ba-991d-37cb69011ca2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:25.334432 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:25.334387 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhbts\" (UniqueName: \"kubernetes.io/projected/c38d4c28-d335-44ba-991d-37cb69011ca2-kube-api-access-dhbts\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:25.334432 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:25.334422 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c38d4c28-d335-44ba-991d-37cb69011ca2-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:25.990528 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:25.990488 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:26.014502 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:26.014465 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfad835c-c46dw" event={"ID":"c38d4c28-d335-44ba-991d-37cb69011ca2","Type":"ContainerDied","Data":"15f1d2ff18977d48ccfe061cbc0714a90af7f866435ff36077b5fbeed7b145e8"} Oct 30 21:39:26.014502 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:26.014494 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfad835c-c46dw" Oct 30 21:39:26.014986 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:26.014545 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15f1d2ff18977d48ccfe061cbc0714a90af7f866435ff36077b5fbeed7b145e8" Oct 30 21:39:26.756971 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:26.756930 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:27.449278 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.449243 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f"] Oct 30 21:39:27.449695 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.449683 5801 cpu_manager.go:414] "containerMap: removing container" podUID="c38d4c28-d335-44ba-991d-37cb69011ca2" containerName="revoke-keys" Oct 30 21:39:27.449737 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.449696 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c38d4c28-d335-44ba-991d-37cb69011ca2" containerName="revoke-keys" Oct 30 21:39:27.449737 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.449715 5801 cpu_manager.go:414] "containerMap: removing container" podUID="596fe353-a183-4cb7-9fe7-b4d9cecf90aa" containerName="revoke-keys" Oct 30 21:39:27.449737 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.449720 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="596fe353-a183-4cb7-9fe7-b4d9cecf90aa" containerName="revoke-keys" Oct 30 21:39:27.449737 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.449732 5801 cpu_manager.go:414] "containerMap: removing container" podUID="89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1" containerName="revoke-keys" Oct 30 21:39:27.449737 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.449737 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1" containerName="revoke-keys" Oct 30 21:39:27.449910 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.449773 5801 cpu_manager.go:414] "containerMap: removing container" podUID="e18b72ce-2635-4754-9cfb-880e5970a807" containerName="revoke-keys" Oct 30 21:39:27.449910 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.449779 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e18b72ce-2635-4754-9cfb-880e5970a807" containerName="revoke-keys" Oct 30 21:39:27.449910 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.449847 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="e18b72ce-2635-4754-9cfb-880e5970a807" containerName="revoke-keys" Oct 30 21:39:27.449910 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.449857 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="c38d4c28-d335-44ba-991d-37cb69011ca2" containerName="revoke-keys" Oct 30 21:39:27.449910 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.449864 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="596fe353-a183-4cb7-9fe7-b4d9cecf90aa" containerName="revoke-keys" Oct 30 21:39:27.449910 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.449872 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1" containerName="revoke-keys" Oct 30 21:39:27.454616 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.454591 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:27.465237 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.464246 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0fa864\"/\"distinct-w4-0fa864-kserve-self-signed-certs\"" Oct 30 21:39:27.466363 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.466338 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f"] Oct 30 21:39:27.513064 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.513025 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:27.561975 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.561939 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0b6e40-2877-48b5-9dba-d3e733b8471d-tls-certs\") pod \"distinct-w4-0fa864-kserve-5977678676-pzd8f\" (UID: \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\") " pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:27.562221 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.561998 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srtvj\" (UniqueName: \"kubernetes.io/projected/ab0b6e40-2877-48b5-9dba-d3e733b8471d-kube-api-access-srtvj\") pod \"distinct-w4-0fa864-kserve-5977678676-pzd8f\" (UID: \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\") " pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:27.562221 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.562115 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-home\") pod \"distinct-w4-0fa864-kserve-5977678676-pzd8f\" (UID: \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\") " pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:27.562221 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.562167 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-model-cache\") pod \"distinct-w4-0fa864-kserve-5977678676-pzd8f\" (UID: \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\") " pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:27.562392 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.562343 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-tmp-dir\") pod \"distinct-w4-0fa864-kserve-5977678676-pzd8f\" (UID: \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\") " pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:27.562392 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.562382 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-dshm\") pod \"distinct-w4-0fa864-kserve-5977678676-pzd8f\" (UID: \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\") " pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:27.616489 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.616447 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:27.663606 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.663576 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-tmp-dir\") pod \"distinct-w4-0fa864-kserve-5977678676-pzd8f\" (UID: \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\") " pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:27.663957 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.663619 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-dshm\") pod \"distinct-w4-0fa864-kserve-5977678676-pzd8f\" (UID: \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\") " pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:27.663957 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.663702 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0b6e40-2877-48b5-9dba-d3e733b8471d-tls-certs\") pod \"distinct-w4-0fa864-kserve-5977678676-pzd8f\" (UID: \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\") " pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:27.663957 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.663766 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srtvj\" (UniqueName: \"kubernetes.io/projected/ab0b6e40-2877-48b5-9dba-d3e733b8471d-kube-api-access-srtvj\") pod \"distinct-w4-0fa864-kserve-5977678676-pzd8f\" (UID: \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\") " pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:27.663957 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.663805 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-home\") pod \"distinct-w4-0fa864-kserve-5977678676-pzd8f\" (UID: \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\") " pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:27.663957 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.663839 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-model-cache\") pod \"distinct-w4-0fa864-kserve-5977678676-pzd8f\" (UID: \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\") " pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:27.664204 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.664067 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-tmp-dir\") pod \"distinct-w4-0fa864-kserve-5977678676-pzd8f\" (UID: \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\") " pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:27.664261 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.664220 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-model-cache\") pod \"distinct-w4-0fa864-kserve-5977678676-pzd8f\" (UID: \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\") " pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:27.664364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.664344 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-home\") pod \"distinct-w4-0fa864-kserve-5977678676-pzd8f\" (UID: \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\") " pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:27.666793 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.666739 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-dshm\") pod \"distinct-w4-0fa864-kserve-5977678676-pzd8f\" (UID: \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\") " pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:27.670231 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.670206 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0b6e40-2877-48b5-9dba-d3e733b8471d-tls-certs\") pod \"distinct-w4-0fa864-kserve-5977678676-pzd8f\" (UID: \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\") " pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:27.693800 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.690337 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srtvj\" (UniqueName: \"kubernetes.io/projected/ab0b6e40-2877-48b5-9dba-d3e733b8471d-kube-api-access-srtvj\") pod \"distinct-w4-0fa864-kserve-5977678676-pzd8f\" (UID: \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\") " pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:27.791425 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:27.791318 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:28.078952 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:28.078925 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f"] Oct 30 21:39:28.079963 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:39:28.079937 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab0b6e40_2877_48b5_9dba_d3e733b8471d.slice/crio-744ff8c8b4f08f064b08e04c46d4227e7f6e1050f94b16ed3390ef564ab6e1b4 WatchSource:0}: Error finding container 744ff8c8b4f08f064b08e04c46d4227e7f6e1050f94b16ed3390ef564ab6e1b4: Status 404 returned error can't find the container with id 744ff8c8b4f08f064b08e04c46d4227e7f6e1050f94b16ed3390ef564ab6e1b4 Oct 30 21:39:28.413697 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:28.413594 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:29.035995 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:29.035962 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" event={"ID":"ab0b6e40-2877-48b5-9dba-d3e733b8471d","Type":"ContainerStarted","Data":"c67da18cae2a5adcdb13dd347588da5a2def36174a2202e3158a4b77bbda2798"} Oct 30 21:39:29.036444 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:29.036001 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" event={"ID":"ab0b6e40-2877-48b5-9dba-d3e733b8471d","Type":"ContainerStarted","Data":"744ff8c8b4f08f064b08e04c46d4227e7f6e1050f94b16ed3390ef564ab6e1b4"} Oct 30 21:39:29.059204 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:29.059155 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podStartSLOduration=2.059141126 podStartE2EDuration="2.059141126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:29.05706553 +0000 UTC m=+637.652882189" watchObservedRunningTime="2026-10-30 21:39:29.059141126 +0000 UTC m=+637.654957785" Oct 30 21:39:29.437110 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:29.437069 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:29.791928 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:29.791838 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:30.448777 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:39:30.447966 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:39:30.450146 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:30.450117 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:31.026777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:30.991848 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:31.039670 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:31.039566 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:31.206926 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:31.206883 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:31.365774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:31.364797 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4665d99f-qbb5m"] Oct 30 21:39:31.416602 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:31.416561 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4665d99f-qbb5m"] Oct 30 21:39:31.416976 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:31.416941 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4665d99f-qbb5m" Oct 30 21:39:31.428227 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:31.427962 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:39:31.428227 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:31.428085 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:39:31.428227 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:31.428198 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:39:31.437780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:31.428932 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:39:31.437780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:31.436741 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwm4z\" (UniqueName: \"kubernetes.io/projected/6512e893-ad04-45d9-bdd9-63bf8a5aec2f-kube-api-access-bwm4z\") pod \"maas-api-revoke-sub-4665d99f-qbb5m\" (UID: \"6512e893-ad04-45d9-bdd9-63bf8a5aec2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4665d99f-qbb5m" Oct 30 21:39:31.437780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:31.436918 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6512e893-ad04-45d9-bdd9-63bf8a5aec2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4665d99f-qbb5m\" (UID: \"6512e893-ad04-45d9-bdd9-63bf8a5aec2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4665d99f-qbb5m" Oct 30 21:39:31.437780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:31.436986 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6512e893-ad04-45d9-bdd9-63bf8a5aec2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4665d99f-qbb5m\" (UID: \"6512e893-ad04-45d9-bdd9-63bf8a5aec2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4665d99f-qbb5m" Oct 30 21:39:31.590126 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:31.590088 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6512e893-ad04-45d9-bdd9-63bf8a5aec2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4665d99f-qbb5m\" (UID: \"6512e893-ad04-45d9-bdd9-63bf8a5aec2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4665d99f-qbb5m" Oct 30 21:39:31.590779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:31.590734 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6512e893-ad04-45d9-bdd9-63bf8a5aec2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4665d99f-qbb5m\" (UID: \"6512e893-ad04-45d9-bdd9-63bf8a5aec2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4665d99f-qbb5m" Oct 30 21:39:31.590974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:31.590953 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwm4z\" (UniqueName: \"kubernetes.io/projected/6512e893-ad04-45d9-bdd9-63bf8a5aec2f-kube-api-access-bwm4z\") pod \"maas-api-revoke-sub-4665d99f-qbb5m\" (UID: \"6512e893-ad04-45d9-bdd9-63bf8a5aec2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4665d99f-qbb5m" Oct 30 21:39:31.592828 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:31.592805 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6512e893-ad04-45d9-bdd9-63bf8a5aec2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4665d99f-qbb5m\" (UID: \"6512e893-ad04-45d9-bdd9-63bf8a5aec2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4665d99f-qbb5m" Oct 30 21:39:31.643325 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:31.642476 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwm4z\" (UniqueName: \"kubernetes.io/projected/6512e893-ad04-45d9-bdd9-63bf8a5aec2f-kube-api-access-bwm4z\") pod \"maas-api-revoke-sub-4665d99f-qbb5m\" (UID: \"6512e893-ad04-45d9-bdd9-63bf8a5aec2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4665d99f-qbb5m" Oct 30 21:39:31.643325 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:31.642849 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6512e893-ad04-45d9-bdd9-63bf8a5aec2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4665d99f-qbb5m\" (UID: \"6512e893-ad04-45d9-bdd9-63bf8a5aec2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4665d99f-qbb5m" Oct 30 21:39:31.821848 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:31.821804 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4665d99f-qbb5m" Oct 30 21:39:32.049845 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:32.046196 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:32.436466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:32.436415 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4665d99f-qbb5m"] Oct 30 21:39:32.437185 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:39:32.437151 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6512e893_ad04_45d9_bdd9_63bf8a5aec2f.slice/crio-4f1dd3992e7956b22eead55bb0334cfc534cbcc74993f757cf7cedfd3c0df01b WatchSource:0}: Error finding container 4f1dd3992e7956b22eead55bb0334cfc534cbcc74993f757cf7cedfd3c0df01b: Status 404 returned error can't find the container with id 4f1dd3992e7956b22eead55bb0334cfc534cbcc74993f757cf7cedfd3c0df01b Oct 30 21:39:33.151005 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:33.150967 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:33.263076 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:33.263040 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4665d99f-qbb5m" event={"ID":"6512e893-ad04-45d9-bdd9-63bf8a5aec2f","Type":"ContainerStarted","Data":"8b6eb94a7b8444c353c1dd66b05caba0226aad28378f670c36cd83672a73a0d2"} Oct 30 21:39:33.263353 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:33.263327 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4665d99f-qbb5m" event={"ID":"6512e893-ad04-45d9-bdd9-63bf8a5aec2f","Type":"ContainerStarted","Data":"4f1dd3992e7956b22eead55bb0334cfc534cbcc74993f757cf7cedfd3c0df01b"} Oct 30 21:39:33.291788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:33.291716 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4665d99f-qbb5m" podStartSLOduration=2.291698283 podStartE2EDuration="2.291698283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:33.281896296 +0000 UTC m=+641.877712961" watchObservedRunningTime="2026-10-30 21:39:33.291698283 +0000 UTC m=+641.887514943" Oct 30 21:39:33.516181 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:39:33.516152 5801 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6512e893_ad04_45d9_bdd9_63bf8a5aec2f.slice/crio-8b6eb94a7b8444c353c1dd66b05caba0226aad28378f670c36cd83672a73a0d2.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:39:33.804672 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:33.804636 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:34.061214 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:34.061133 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:34.271612 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:34.271576 5801 generic.go:356] "Generic (PLEG): container finished" podID="6512e893-ad04-45d9-bdd9-63bf8a5aec2f" containerID="8b6eb94a7b8444c353c1dd66b05caba0226aad28378f670c36cd83672a73a0d2" exitCode=0 Oct 30 21:39:34.272073 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:34.271652 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4665d99f-qbb5m" event={"ID":"6512e893-ad04-45d9-bdd9-63bf8a5aec2f","Type":"ContainerDied","Data":"8b6eb94a7b8444c353c1dd66b05caba0226aad28378f670c36cd83672a73a0d2"} Oct 30 21:39:34.496826 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:34.496736 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a86bb3c6-r4pzd"] Oct 30 21:39:34.500848 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:34.500830 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86bb3c6-r4pzd" Oct 30 21:39:34.509967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:34.509946 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a86bb3c6-r4pzd"] Oct 30 21:39:34.615180 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:34.615148 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b503661-236a-483a-9395-53a63284643e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a86bb3c6-r4pzd\" (UID: \"2b503661-236a-483a-9395-53a63284643e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86bb3c6-r4pzd" Oct 30 21:39:34.615362 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:34.615339 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b503661-236a-483a-9395-53a63284643e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a86bb3c6-r4pzd\" (UID: \"2b503661-236a-483a-9395-53a63284643e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86bb3c6-r4pzd" Oct 30 21:39:34.615456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:34.615444 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsl7g\" (UniqueName: \"kubernetes.io/projected/2b503661-236a-483a-9395-53a63284643e-kube-api-access-qsl7g\") pod \"maas-api-revoke-sub-a86bb3c6-r4pzd\" (UID: \"2b503661-236a-483a-9395-53a63284643e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86bb3c6-r4pzd" Oct 30 21:39:34.716387 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:34.716357 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b503661-236a-483a-9395-53a63284643e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a86bb3c6-r4pzd\" (UID: \"2b503661-236a-483a-9395-53a63284643e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86bb3c6-r4pzd" Oct 30 21:39:34.716531 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:34.716422 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsl7g\" (UniqueName: \"kubernetes.io/projected/2b503661-236a-483a-9395-53a63284643e-kube-api-access-qsl7g\") pod \"maas-api-revoke-sub-a86bb3c6-r4pzd\" (UID: \"2b503661-236a-483a-9395-53a63284643e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86bb3c6-r4pzd" Oct 30 21:39:34.716531 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:34.716450 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b503661-236a-483a-9395-53a63284643e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a86bb3c6-r4pzd\" (UID: \"2b503661-236a-483a-9395-53a63284643e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86bb3c6-r4pzd" Oct 30 21:39:34.717106 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:34.717078 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b503661-236a-483a-9395-53a63284643e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a86bb3c6-r4pzd\" (UID: \"2b503661-236a-483a-9395-53a63284643e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86bb3c6-r4pzd" Oct 30 21:39:34.718644 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:34.718626 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b503661-236a-483a-9395-53a63284643e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a86bb3c6-r4pzd\" (UID: \"2b503661-236a-483a-9395-53a63284643e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86bb3c6-r4pzd" Oct 30 21:39:34.731544 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:34.731516 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsl7g\" (UniqueName: \"kubernetes.io/projected/2b503661-236a-483a-9395-53a63284643e-kube-api-access-qsl7g\") pod \"maas-api-revoke-sub-a86bb3c6-r4pzd\" (UID: \"2b503661-236a-483a-9395-53a63284643e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86bb3c6-r4pzd" Oct 30 21:39:34.815863 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:34.815836 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86bb3c6-r4pzd" Oct 30 21:39:34.835023 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:34.834893 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:35.094078 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:35.094047 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a86bb3c6-r4pzd"] Oct 30 21:39:35.094846 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:39:35.094812 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b503661_236a_483a_9395_53a63284643e.slice/crio-74f812b3c75f74c02625f1b51b17b6b0745eb0e27299ab2f9ff65918e732af9b WatchSource:0}: Error finding container 74f812b3c75f74c02625f1b51b17b6b0745eb0e27299ab2f9ff65918e732af9b: Status 404 returned error can't find the container with id 74f812b3c75f74c02625f1b51b17b6b0745eb0e27299ab2f9ff65918e732af9b Oct 30 21:39:35.279560 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:35.279527 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86bb3c6-r4pzd" event={"ID":"2b503661-236a-483a-9395-53a63284643e","Type":"ContainerStarted","Data":"74f812b3c75f74c02625f1b51b17b6b0745eb0e27299ab2f9ff65918e732af9b"} Oct 30 21:39:35.403423 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:35.403401 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4665d99f-qbb5m" Oct 30 21:39:35.527298 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:35.527272 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6512e893-ad04-45d9-bdd9-63bf8a5aec2f-kube-api-access-bwm4z\" (UniqueName: \"kubernetes.io/projected/6512e893-ad04-45d9-bdd9-63bf8a5aec2f-kube-api-access-bwm4z\") pod \"6512e893-ad04-45d9-bdd9-63bf8a5aec2f\" (UID: \"6512e893-ad04-45d9-bdd9-63bf8a5aec2f\") " Oct 30 21:39:35.527532 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:35.527336 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6512e893-ad04-45d9-bdd9-63bf8a5aec2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6512e893-ad04-45d9-bdd9-63bf8a5aec2f-maas-api-xks-ca\") pod \"6512e893-ad04-45d9-bdd9-63bf8a5aec2f\" (UID: \"6512e893-ad04-45d9-bdd9-63bf8a5aec2f\") " Oct 30 21:39:35.527532 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:35.527502 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6512e893-ad04-45d9-bdd9-63bf8a5aec2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6512e893-ad04-45d9-bdd9-63bf8a5aec2f-maas-api-openshift-ca\") pod \"6512e893-ad04-45d9-bdd9-63bf8a5aec2f\" (UID: \"6512e893-ad04-45d9-bdd9-63bf8a5aec2f\") " Oct 30 21:39:35.527896 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:35.527871 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6512e893-ad04-45d9-bdd9-63bf8a5aec2f-maas-api-openshift-ca" pod "6512e893-ad04-45d9-bdd9-63bf8a5aec2f" (UID: "6512e893-ad04-45d9-bdd9-63bf8a5aec2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:35.529364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:35.529340 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6512e893-ad04-45d9-bdd9-63bf8a5aec2f-maas-api-xks-ca" pod "6512e893-ad04-45d9-bdd9-63bf8a5aec2f" (UID: "6512e893-ad04-45d9-bdd9-63bf8a5aec2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:35.529850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:35.529832 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6512e893-ad04-45d9-bdd9-63bf8a5aec2f-kube-api-access-bwm4z" pod "6512e893-ad04-45d9-bdd9-63bf8a5aec2f" (UID: "6512e893-ad04-45d9-bdd9-63bf8a5aec2f"). InnerVolumeSpecName "kube-api-access-bwm4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:35.558938 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:35.558906 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:35.629348 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:35.629259 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6512e893-ad04-45d9-bdd9-63bf8a5aec2f-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:35.629348 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:35.629299 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwm4z\" (UniqueName: \"kubernetes.io/projected/6512e893-ad04-45d9-bdd9-63bf8a5aec2f-kube-api-access-bwm4z\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:35.629348 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:35.629317 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6512e893-ad04-45d9-bdd9-63bf8a5aec2f-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:36.289057 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:36.288974 5801 generic.go:356] "Generic (PLEG): container finished" podID="2b503661-236a-483a-9395-53a63284643e" containerID="6fb385f66bcf153f8fa461795f5eea0abbc26a8931d7fd8b94501aa42eb33b2b" exitCode=0 Oct 30 21:39:36.289409 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:36.289055 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86bb3c6-r4pzd" event={"ID":"2b503661-236a-483a-9395-53a63284643e","Type":"ContainerDied","Data":"6fb385f66bcf153f8fa461795f5eea0abbc26a8931d7fd8b94501aa42eb33b2b"} Oct 30 21:39:36.290472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:36.290453 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4665d99f-qbb5m" Oct 30 21:39:36.290595 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:36.290487 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4665d99f-qbb5m" event={"ID":"6512e893-ad04-45d9-bdd9-63bf8a5aec2f","Type":"ContainerDied","Data":"4f1dd3992e7956b22eead55bb0334cfc534cbcc74993f757cf7cedfd3c0df01b"} Oct 30 21:39:36.290595 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:36.290522 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f1dd3992e7956b22eead55bb0334cfc534cbcc74993f757cf7cedfd3c0df01b" Oct 30 21:39:36.320319 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:36.320295 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:37.147322 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:37.147267 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:37.439880 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:37.439857 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86bb3c6-r4pzd" Oct 30 21:39:37.552204 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:37.552175 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b503661-236a-483a-9395-53a63284643e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b503661-236a-483a-9395-53a63284643e-maas-api-xks-ca\") pod \"2b503661-236a-483a-9395-53a63284643e\" (UID: \"2b503661-236a-483a-9395-53a63284643e\") " Oct 30 21:39:37.552337 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:37.552225 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b503661-236a-483a-9395-53a63284643e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b503661-236a-483a-9395-53a63284643e-maas-api-openshift-ca\") pod \"2b503661-236a-483a-9395-53a63284643e\" (UID: \"2b503661-236a-483a-9395-53a63284643e\") " Oct 30 21:39:37.552337 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:37.552283 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b503661-236a-483a-9395-53a63284643e-kube-api-access-qsl7g\" (UniqueName: \"kubernetes.io/projected/2b503661-236a-483a-9395-53a63284643e-kube-api-access-qsl7g\") pod \"2b503661-236a-483a-9395-53a63284643e\" (UID: \"2b503661-236a-483a-9395-53a63284643e\") " Oct 30 21:39:37.552617 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:37.552594 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b503661-236a-483a-9395-53a63284643e-maas-api-openshift-ca" pod "2b503661-236a-483a-9395-53a63284643e" (UID: "2b503661-236a-483a-9395-53a63284643e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:37.552714 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:37.552699 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b503661-236a-483a-9395-53a63284643e-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:37.554315 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:37.554292 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b503661-236a-483a-9395-53a63284643e-maas-api-xks-ca" pod "2b503661-236a-483a-9395-53a63284643e" (UID: "2b503661-236a-483a-9395-53a63284643e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:37.554415 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:37.554369 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b503661-236a-483a-9395-53a63284643e-kube-api-access-qsl7g" pod "2b503661-236a-483a-9395-53a63284643e" (UID: "2b503661-236a-483a-9395-53a63284643e"). InnerVolumeSpecName "kube-api-access-qsl7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:37.653975 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:37.653944 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b503661-236a-483a-9395-53a63284643e-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:37.654095 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:37.653980 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsl7g\" (UniqueName: \"kubernetes.io/projected/2b503661-236a-483a-9395-53a63284643e-kube-api-access-qsl7g\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:37.791589 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:37.791516 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:37.804242 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:37.804218 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:38.024943 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.024903 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:38.132143 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.132112 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd"] Oct 30 21:39:38.132607 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.132585 5801 cpu_manager.go:414] "containerMap: removing container" podUID="2b503661-236a-483a-9395-53a63284643e" containerName="revoke-keys" Oct 30 21:39:38.132607 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.132606 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b503661-236a-483a-9395-53a63284643e" containerName="revoke-keys" Oct 30 21:39:38.132779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.132642 5801 cpu_manager.go:414] "containerMap: removing container" podUID="6512e893-ad04-45d9-bdd9-63bf8a5aec2f" containerName="revoke-keys" Oct 30 21:39:38.132779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.132650 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6512e893-ad04-45d9-bdd9-63bf8a5aec2f" containerName="revoke-keys" Oct 30 21:39:38.132779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.132745 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b503661-236a-483a-9395-53a63284643e" containerName="revoke-keys" Oct 30 21:39:38.132779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.132771 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="6512e893-ad04-45d9-bdd9-63bf8a5aec2f" containerName="revoke-keys" Oct 30 21:39:38.136279 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.136260 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.151074 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.151050 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd"] Oct 30 21:39:38.259406 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.259368 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnbxs\" (UniqueName: \"kubernetes.io/projected/9a837073-7227-4ae9-9338-a70202213538-kube-api-access-wnbxs\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.259406 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.259406 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a837073-7227-4ae9-9338-a70202213538-istio-token\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.259648 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.259427 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a837073-7227-4ae9-9338-a70202213538-istiod-ca-cert\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.259648 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.259507 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-credential-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.259648 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.259554 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-istio-envoy\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.259744 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.259733 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a837073-7227-4ae9-9338-a70202213538-istio-podinfo\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.259820 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.259779 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-workload-certs\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.259876 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.259860 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-workload-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.259934 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.259912 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-istio-data\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.304584 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.304557 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86bb3c6-r4pzd" Oct 30 21:39:38.304735 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.304592 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86bb3c6-r4pzd" event={"ID":"2b503661-236a-483a-9395-53a63284643e","Type":"ContainerDied","Data":"74f812b3c75f74c02625f1b51b17b6b0745eb0e27299ab2f9ff65918e732af9b"} Oct 30 21:39:38.304735 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.304638 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74f812b3c75f74c02625f1b51b17b6b0745eb0e27299ab2f9ff65918e732af9b" Oct 30 21:39:38.316088 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.316057 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:38.361040 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.361013 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-workload-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.361171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.361048 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-istio-data\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.361171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.361108 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnbxs\" (UniqueName: \"kubernetes.io/projected/9a837073-7227-4ae9-9338-a70202213538-kube-api-access-wnbxs\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.361171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.361126 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a837073-7227-4ae9-9338-a70202213538-istio-token\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.361171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.361148 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a837073-7227-4ae9-9338-a70202213538-istiod-ca-cert\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.361400 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.361178 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-credential-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.361400 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.361195 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-istio-envoy\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.361400 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.361282 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a837073-7227-4ae9-9338-a70202213538-istio-podinfo\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.361400 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.361309 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-workload-certs\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.361665 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.361458 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-workload-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.361665 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.361634 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-workload-certs\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.362348 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.362326 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-credential-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.362500 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.362330 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a837073-7227-4ae9-9338-a70202213538-istiod-ca-cert\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.362500 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.362383 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-istio-data\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.364429 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.364298 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-istio-envoy\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.364724 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.364698 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a837073-7227-4ae9-9338-a70202213538-istio-podinfo\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.374183 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.374154 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnbxs\" (UniqueName: \"kubernetes.io/projected/9a837073-7227-4ae9-9338-a70202213538-kube-api-access-wnbxs\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.374349 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.374329 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a837073-7227-4ae9-9338-a70202213538-istio-token\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.451815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.451712 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:38.756876 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.756844 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd"] Oct 30 21:39:38.757137 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:39:38.757119 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a837073_7227_4ae9_9338_a70202213538.slice/crio-4346bf01dca7c0cccd45ad822bf394bfc25ef10471a21e9d5c3b3e7dc20a8665 WatchSource:0}: Error finding container 4346bf01dca7c0cccd45ad822bf394bfc25ef10471a21e9d5c3b3e7dc20a8665: Status 404 returned error can't find the container with id 4346bf01dca7c0cccd45ad822bf394bfc25ef10471a21e9d5c3b3e7dc20a8665 Oct 30 21:39:38.759626 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.759590 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:39:38.759702 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.759657 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:39:38.759702 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.759689 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:39:38.869250 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:38.869195 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:39.314814 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:39.314773 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" event={"ID":"9a837073-7227-4ae9-9338-a70202213538","Type":"ContainerStarted","Data":"bd461a09ddfde0ed4c2332971e30b94c12f1ecb322523dd7ad8d581e0ad2cbc3"} Oct 30 21:39:39.315001 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:39.314839 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" event={"ID":"9a837073-7227-4ae9-9338-a70202213538","Type":"ContainerStarted","Data":"4346bf01dca7c0cccd45ad822bf394bfc25ef10471a21e9d5c3b3e7dc20a8665"} Oct 30 21:39:39.342971 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:39.342915 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" podStartSLOduration=1.342897214 podStartE2EDuration="1.342897214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:39.340149734 +0000 UTC m=+647.935966384" watchObservedRunningTime="2026-10-30 21:39:39.342897214 +0000 UTC m=+647.938713871" Oct 30 21:39:39.452177 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:39.452139 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:39.480776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:39.480584 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 21:39:39.480776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:39.480647 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" podUID="9a837073-7227-4ae9-9338-a70202213538" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 21:39:39.829403 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:39.829367 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:40.452426 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:40.452392 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 21:39:40.452856 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:40.452450 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" podUID="9a837073-7227-4ae9-9338-a70202213538" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 21:39:40.805806 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:40.805747 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:41.132207 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:41.132111 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:41.230838 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:41.230798 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:41.452518 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:41.452434 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 21:39:41.452518 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:41.452492 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" podUID="9a837073-7227-4ae9-9338-a70202213538" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 21:39:41.978553 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:41.978255 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:42.452307 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:42.452272 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 21:39:42.452455 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:42.452328 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" podUID="9a837073-7227-4ae9-9338-a70202213538" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 21:39:43.078501 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:43.078465 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:43.452166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:43.452085 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 21:39:43.452166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:43.452136 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" podUID="9a837073-7227-4ae9-9338-a70202213538" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 21:39:44.330488 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:44.330450 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:44.452615 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:44.452581 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 21:39:44.452798 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:44.452654 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" podUID="9a837073-7227-4ae9-9338-a70202213538" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 21:39:44.528597 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:44.528552 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:45.454791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:45.454738 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 21:39:45.455375 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:45.455341 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" podUID="9a837073-7227-4ae9-9338-a70202213538" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 21:39:46.517640 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:46.517603 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:46.624424 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:46.624389 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:47.020570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:47.020538 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:47.021014 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:47.020990 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:47.021916 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:47.021894 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:39:47.546154 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:47.546095 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:47.850635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:47.850593 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:48.633496 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:48.633445 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:49.433415 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:49.433380 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:50.132421 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:50.132385 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:50.238124 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:50.238090 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:50.856770 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:50.856717 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:51.650521 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:51.650486 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:52.334271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:52.334231 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:53.131201 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.131165 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:53.553375 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.553347 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk"] Oct 30 21:39:53.559811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.559735 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.573367 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.573337 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk"] Oct 30 21:39:53.593436 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.593407 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f"] Oct 30 21:39:53.597452 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.597431 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.613416 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.613391 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f"] Oct 30 21:39:53.683655 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.683615 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r"] Oct 30 21:39:53.687730 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.687708 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.689860 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.689829 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.689957 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.689871 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-data\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.689957 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.689891 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-token\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.690022 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.689967 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.690131 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.690112 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.690179 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.690163 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.690250 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.690235 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.690345 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.690327 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft6jp\" (UniqueName: \"kubernetes.io/projected/e3976e0e-4921-46d0-99f4-2c26733ef15a-kube-api-access-ft6jp\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.690397 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.690381 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-podinfo\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.690452 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.690443 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.690488 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.690470 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-workload-certs\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.690534 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.690485 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkxck\" (UniqueName: \"kubernetes.io/projected/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-kube-api-access-nkxck\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.690534 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.690521 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-envoy\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.690635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.690623 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.690667 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.690652 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3976e0e-4921-46d0-99f4-2c26733ef15a-istiod-ca-cert\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.690701 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.690693 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-workload-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.690737 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.690714 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.690875 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.690854 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-credential-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.700139 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.700117 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r"] Oct 30 21:39:53.791526 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.791486 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-envoy\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.791693 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.791532 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.791693 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.791570 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3976e0e-4921-46d0-99f4-2c26733ef15a-istiod-ca-cert\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.791693 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.791607 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-workload-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.791906 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.791814 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.791906 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.791865 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-credential-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.791906 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.791899 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-podinfo\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.792069 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.791923 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-credential-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.792069 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.791948 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.792069 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.792018 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-data\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.792069 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.792044 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-workload-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.792280 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.792074 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-token\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.792280 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.792224 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.792387 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.792286 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.792387 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.792299 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-credential-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.792387 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.792290 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.792387 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.792351 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-data\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.792387 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.792352 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-data\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.792681 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.792473 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3976e0e-4921-46d0-99f4-2c26733ef15a-istiod-ca-cert\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.792681 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.792509 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-workload-certs\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.792681 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.792630 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.792965 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.792731 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.792965 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.792783 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.792965 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.792834 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-workload-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.792965 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.792873 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-envoy\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.792965 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.792902 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft6jp\" (UniqueName: \"kubernetes.io/projected/e3976e0e-4921-46d0-99f4-2c26733ef15a-kube-api-access-ft6jp\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.793187 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.792983 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-podinfo\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.793187 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.793023 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.793187 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.793048 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istiod-ca-cert\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.793187 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.793093 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-workload-certs\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.793503 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.793467 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkxck\" (UniqueName: \"kubernetes.io/projected/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-kube-api-access-nkxck\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.793608 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.793520 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-token\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.793797 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.793773 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-workload-certs\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.793914 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.793825 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxq25\" (UniqueName: \"kubernetes.io/projected/db7b6603-3dfd-4f20-a362-da44b2fb9da9-kube-api-access-nxq25\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.794067 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.794011 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.794504 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.794484 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-envoy\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.794600 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.794503 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.795162 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.795137 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.796446 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.796415 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.796558 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.796417 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.796683 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.796664 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-podinfo\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.798170 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.798147 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj"] Oct 30 21:39:53.804475 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.804425 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:53.810401 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.810380 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-token\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.810812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.810784 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkxck\" (UniqueName: \"kubernetes.io/projected/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-kube-api-access-nkxck\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.810812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.810861 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft6jp\" (UniqueName: \"kubernetes.io/projected/e3976e0e-4921-46d0-99f4-2c26733ef15a-kube-api-access-ft6jp\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.813912 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.813885 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.819425 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.819403 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj"] Oct 30 21:39:53.880437 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.880417 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:53.895233 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.895205 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-data\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.895345 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.895240 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-envoy\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:53.895345 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.895269 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-workload-certs\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.895452 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.895339 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-credential-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:53.895452 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.895413 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-workload-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.895452 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.895447 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-workload-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:53.895585 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.895523 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-envoy\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.895585 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.895555 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bdabb86-13f1-4ab6-8b08-418fff19b933-istiod-ca-cert\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:53.895684 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.895603 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-data\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.895684 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.895617 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-data\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:53.895684 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.895631 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-workload-certs\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.895844 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.895712 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istiod-ca-cert\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.895844 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.895746 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-workload-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.895844 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.895787 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-token\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.895844 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.895812 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxq25\" (UniqueName: \"kubernetes.io/projected/db7b6603-3dfd-4f20-a362-da44b2fb9da9-kube-api-access-nxq25\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.896039 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.895892 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-podinfo\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:53.896039 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.895918 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvjxf\" (UniqueName: \"kubernetes.io/projected/7bdabb86-13f1-4ab6-8b08-418fff19b933-kube-api-access-lvjxf\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:53.896039 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.896019 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-podinfo\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.896199 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.896054 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-credential-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.896199 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.896087 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-token\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:53.896199 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.896136 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-workload-certs\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:53.896375 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.896352 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istiod-ca-cert\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.896458 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.896439 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-credential-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.897812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.897786 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-envoy\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.898935 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.898918 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-podinfo\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.912803 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.912777 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-token\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.913580 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.913562 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxq25\" (UniqueName: \"kubernetes.io/projected/db7b6603-3dfd-4f20-a362-da44b2fb9da9-kube-api-access-nxq25\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:53.915440 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.915424 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:53.962406 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:53.962249 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:54.006368 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.001958 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-credential-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:54.006368 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.002076 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-workload-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:54.006368 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.002564 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-credential-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:54.006368 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.003511 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-workload-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:54.006368 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.003556 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bdabb86-13f1-4ab6-8b08-418fff19b933-istiod-ca-cert\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:54.006368 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.003677 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-data\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:54.006368 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.003851 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-podinfo\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:54.006368 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.003905 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvjxf\" (UniqueName: \"kubernetes.io/projected/7bdabb86-13f1-4ab6-8b08-418fff19b933-kube-api-access-lvjxf\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:54.006368 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.004037 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-token\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:54.006368 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.004157 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-workload-certs\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:54.006368 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.004197 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-envoy\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:54.008720 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.006427 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-workload-certs\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:54.008720 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.006796 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-data\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:54.008720 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.007862 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:54.008720 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.008085 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bdabb86-13f1-4ab6-8b08-418fff19b933-istiod-ca-cert\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:54.024073 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.023079 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-envoy\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:54.024073 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.023221 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-podinfo\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:54.027308 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.027024 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvjxf\" (UniqueName: \"kubernetes.io/projected/7bdabb86-13f1-4ab6-8b08-418fff19b933-kube-api-access-lvjxf\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:54.027308 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.027279 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-token\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:54.151773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.151732 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:54.169251 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.169224 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk"] Oct 30 21:39:54.169697 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:39:54.169670 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88c55fe_afa3_4b4e_b5b7_e4f8e35e51d7.slice/crio-17417a7b858207a41b0cd6690be4fbf42099c10b30c0febd84d31c6e7cab9984 WatchSource:0}: Error finding container 17417a7b858207a41b0cd6690be4fbf42099c10b30c0febd84d31c6e7cab9984: Status 404 returned error can't find the container with id 17417a7b858207a41b0cd6690be4fbf42099c10b30c0febd84d31c6e7cab9984 Oct 30 21:39:54.172142 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.172111 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:39:54.172238 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.172178 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:39:54.172238 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.172210 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:39:54.227083 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:39:54.225136 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3976e0e_4921_46d0_99f4_2c26733ef15a.slice/crio-9002f93bf2c66761355b5248bab3cf125c76063994ac84384851a88fce83ce76 WatchSource:0}: Error finding container 9002f93bf2c66761355b5248bab3cf125c76063994ac84384851a88fce83ce76: Status 404 returned error can't find the container with id 9002f93bf2c66761355b5248bab3cf125c76063994ac84384851a88fce83ce76 Oct 30 21:39:54.229278 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.229234 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:39:54.229399 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.229368 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:39:54.229468 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.229418 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:39:54.232990 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.231665 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f"] Oct 30 21:39:54.330293 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:39:54.329787 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb7b6603_3dfd_4f20_a362_da44b2fb9da9.slice/crio-d6d154e7a275586ca06112a3271387081e56d14ddf8a6f2f29dcd3374c761b4c WatchSource:0}: Error finding container d6d154e7a275586ca06112a3271387081e56d14ddf8a6f2f29dcd3374c761b4c: Status 404 returned error can't find the container with id d6d154e7a275586ca06112a3271387081e56d14ddf8a6f2f29dcd3374c761b4c Oct 30 21:39:54.333610 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.333574 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:39:54.333705 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.333660 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:39:54.333791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.333706 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:39:54.334094 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.334072 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r"] Oct 30 21:39:54.455129 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.455097 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj"] Oct 30 21:39:54.496544 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.496498 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:39:54.496684 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.496590 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:39:54.496684 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.496629 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:39:54.504040 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.501225 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" event={"ID":"db7b6603-3dfd-4f20-a362-da44b2fb9da9","Type":"ContainerStarted","Data":"d9e6af01d26650702e74d0f2291aa5472f8bc11ab3426247d2970ea56202f1cb"} Oct 30 21:39:54.504040 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.501275 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" event={"ID":"db7b6603-3dfd-4f20-a362-da44b2fb9da9","Type":"ContainerStarted","Data":"d6d154e7a275586ca06112a3271387081e56d14ddf8a6f2f29dcd3374c761b4c"} Oct 30 21:39:54.505093 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.505061 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" event={"ID":"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7","Type":"ContainerStarted","Data":"fc31a7714628ad17f8e7d9b3a1d9f880d34295fce5cb38456fa06155c669ff1b"} Oct 30 21:39:54.505213 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.505114 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" event={"ID":"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7","Type":"ContainerStarted","Data":"17417a7b858207a41b0cd6690be4fbf42099c10b30c0febd84d31c6e7cab9984"} Oct 30 21:39:54.509209 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.509181 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" event={"ID":"e3976e0e-4921-46d0-99f4-2c26733ef15a","Type":"ContainerStarted","Data":"62dc366945d2c550b3e888b0e7d4b2ac39efa9bd3f4952e4310d1eb5ad2daa04"} Oct 30 21:39:54.509302 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.509232 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" event={"ID":"e3976e0e-4921-46d0-99f4-2c26733ef15a","Type":"ContainerStarted","Data":"9002f93bf2c66761355b5248bab3cf125c76063994ac84384851a88fce83ce76"} Oct 30 21:39:54.532500 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.532442 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" podStartSLOduration=1.5324254000000002 podStartE2EDuration="1.5324254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:54.525983396 +0000 UTC m=+663.121800072" watchObservedRunningTime="2026-10-30 21:39:54.5324254 +0000 UTC m=+663.128242061" Oct 30 21:39:54.553706 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.553641 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" podStartSLOduration=1.5536241629999998 podStartE2EDuration="1.553624163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:54.550249118 +0000 UTC m=+663.146065782" watchObservedRunningTime="2026-10-30 21:39:54.553624163 +0000 UTC m=+663.149440825" Oct 30 21:39:54.579330 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.578034 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" podStartSLOduration=1.578020223 podStartE2EDuration="1.578020223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:54.577000711 +0000 UTC m=+663.172817400" watchObservedRunningTime="2026-10-30 21:39:54.578020223 +0000 UTC m=+663.173836880" Oct 30 21:39:54.754775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.754730 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:54.885998 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.885890 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:39:54.916098 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:54.916066 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:39:55.010777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:55.009921 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:39:55.012587 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:55.012547 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:39:55.012842 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:55.012799 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" podUID="db7b6603-3dfd-4f20-a362-da44b2fb9da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 21:39:55.634950 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:55.634913 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" event={"ID":"7bdabb86-13f1-4ab6-8b08-418fff19b933","Type":"ContainerStarted","Data":"9caf6898b09fe61c8823dc54a7712a266992f32063b91d42f5d6d8bdeab947f6"} Oct 30 21:39:55.635550 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:55.635524 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" event={"ID":"7bdabb86-13f1-4ab6-8b08-418fff19b933","Type":"ContainerStarted","Data":"76d450422d5595f476b29903f3567120a24364eab18fa66ed5e5b6f4e020620e"} Oct 30 21:39:55.686134 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:55.682136 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" podStartSLOduration=2.68211563 podStartE2EDuration="2.68211563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:55.675083154 +0000 UTC m=+664.270899840" watchObservedRunningTime="2026-10-30 21:39:55.68211563 +0000 UTC m=+664.277932291" Oct 30 21:39:55.885555 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:55.885459 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:39:55.885829 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:55.885799 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" podUID="b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 21:39:55.948826 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:55.940974 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:39:55.948826 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:55.941040 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" podUID="e3976e0e-4921-46d0-99f4-2c26733ef15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 21:39:56.014303 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:56.014264 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:39:56.014581 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:56.014540 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" podUID="db7b6603-3dfd-4f20-a362-da44b2fb9da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 21:39:56.163967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:56.154197 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:39:56.163967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:56.154361 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:39:56.163967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:56.154410 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" podUID="7bdabb86-13f1-4ab6-8b08-418fff19b933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 21:39:56.260735 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:56.260699 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:56.262110 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:56.262080 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:56.330903 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:56.330855 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:56.348893 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:56.348846 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:56.881269 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:56.881226 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:39:56.881887 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:56.881854 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" podUID="b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 21:39:56.927781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:56.924163 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:39:56.927781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:56.924224 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" podUID="e3976e0e-4921-46d0-99f4-2c26733ef15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 21:39:56.967790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:56.964799 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:39:56.997825 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:56.997206 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:39:57.017779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:57.017133 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:39:57.017779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:57.017195 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" podUID="db7b6603-3dfd-4f20-a362-da44b2fb9da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 21:39:57.043474 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:57.043428 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:57.124132 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:57.124087 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:57.152693 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:57.152174 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:39:57.152693 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:57.152237 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" podUID="7bdabb86-13f1-4ab6-8b08-418fff19b933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 21:39:57.163404 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:57.163363 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:39:57.182850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:57.181996 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:57.275431 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:57.275273 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:57.275784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:57.275733 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 21:39:57.282823 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:57.276518 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:39:57.620858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:57.333147 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:39:57.620858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:57.381865 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:57.620858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:57.491734 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:39:57.620858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:57.491806 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:39:57.620858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:57.493147 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.004046 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.003936 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:39:58.044828 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.009557 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.078778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.077885 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:39:58.093787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.010676 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:39:58.093787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.090361 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" podUID="db7b6603-3dfd-4f20-a362-da44b2fb9da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 21:39:58.093787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.018120 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:39:58.093787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.090425 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" podUID="e3976e0e-4921-46d0-99f4-2c26733ef15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 21:39:58.093787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.069366 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:39:58.093787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.090468 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" podUID="b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 21:39:58.120099 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.119996 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:58.121542 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.121514 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:39:58.178144 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.178114 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:39:58.178431 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.178414 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:39:58.180224 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.180202 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:39:58.183777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.127655 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.184065 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.184034 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.205811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.176001 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.205811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.198875 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:39:58.205811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.199033 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" podUID="7bdabb86-13f1-4ab6-8b08-418fff19b933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 21:39:58.205811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.198951 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:39:58.205811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.199892 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:39:58.205811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.199953 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:39:58.289834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.279788 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 21:39:58.534159 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.496906 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:39:58.534159 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.498205 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:39:58.534159 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.498496 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:39:58.534159 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.498733 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:39:58.566775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.559028 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.832827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.832109 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.886775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.886396 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:39:58.886775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.886451 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" podUID="b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 21:39:58.925912 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:58.925875 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:39:59.081775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:59.061047 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" podUID="e3976e0e-4921-46d0-99f4-2c26733ef15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 21:39:59.081775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:59.058834 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:39:59.081775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:59.010138 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:39:59.081775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:59.061774 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" podUID="db7b6603-3dfd-4f20-a362-da44b2fb9da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 21:39:59.108830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:59.083606 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:59.185374 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:59.185335 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:39:59.185655 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:59.185626 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" podUID="7bdabb86-13f1-4ab6-8b08-418fff19b933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 21:39:59.186276 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:59.087173 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:59.186531 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:59.186509 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:59.187364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:59.187338 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:39:59.231778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:59.228227 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:59.323865 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:59.318916 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Oct 30 21:39:59.426124 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:59.348848 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 21:39:59.508920 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:59.508872 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:59.531779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:59.509507 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:39:59.532689 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:59.521462 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:39:59.890994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:59.890783 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:39:59.891204 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:59.891148 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" podUID="b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 21:39:59.921910 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:59.921871 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:39:59.922460 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:39:59.922427 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" podUID="e3976e0e-4921-46d0-99f4-2c26733ef15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 21:40:00.026904 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:00.026866 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:40:00.027412 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:00.027380 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" podUID="db7b6603-3dfd-4f20-a362-da44b2fb9da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 21:40:00.069889 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:00.069830 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:00.079778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:00.077847 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:00.144954 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:00.144843 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:40:00.153322 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:00.153288 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:40:00.153587 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:00.153556 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" podUID="7bdabb86-13f1-4ab6-8b08-418fff19b933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 21:40:00.186917 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:00.186867 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:00.189903 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:00.188566 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:00.232470 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:00.232413 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:00.273206 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:00.273159 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:00.430292 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:00.429060 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:00.440277 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:40:00.439912 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:40:00.518776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:00.518477 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:00.544596 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:00.544554 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:00.545205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:00.544837 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:00.909778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:00.905910 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:40:00.909778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:00.905977 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" podUID="b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 21:40:00.935677 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:00.935642 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:40:00.936038 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:00.935975 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" podUID="e3976e0e-4921-46d0-99f4-2c26733ef15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 21:40:01.023020 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:01.022534 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:40:01.023020 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:01.022604 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" podUID="db7b6603-3dfd-4f20-a362-da44b2fb9da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 21:40:01.065657 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:01.065549 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:01.070564 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:01.070485 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:01.082787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:01.080836 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:01.082787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:01.081245 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:40:01.155904 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:01.155857 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:01.188510 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:01.187856 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:40:01.223590 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:01.222865 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:01.229000 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:01.228967 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:40:01.304924 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:01.301142 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" podUID="7bdabb86-13f1-4ab6-8b08-418fff19b933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 21:40:01.304924 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:01.233848 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:01.387777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:01.384415 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:01.434898 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:01.434849 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:01.534109 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:01.534008 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:01.576832 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:01.560842 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:01.576832 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:01.560888 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:40:01.884788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:01.881086 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:40:01.884788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:01.881140 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" podUID="b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 21:40:01.916421 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:01.916362 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:40:01.916634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:01.916426 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" podUID="e3976e0e-4921-46d0-99f4-2c26733ef15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 21:40:02.010229 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.010187 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:40:02.010502 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.010476 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" podUID="db7b6603-3dfd-4f20-a362-da44b2fb9da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 21:40:02.071883 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.071839 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:40:02.103087 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.103043 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:02.143375 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.143264 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:02.152150 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.152117 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:40:02.152416 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.152389 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" podUID="7bdabb86-13f1-4ab6-8b08-418fff19b933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 21:40:02.158920 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.158847 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:02.190851 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.190802 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:02.226346 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.225798 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:40:02.268054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.268013 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded" Oct 30 21:40:02.312990 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.312931 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:40:02.437424 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.437320 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:02.579942 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.579893 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:02.580298 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.580156 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:40:02.580897 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.580181 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:40:02.623624 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.623584 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:02.735778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.735624 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:40:02.783081 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.783046 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:40:02.816154 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.815026 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:40:02.875317 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.868905 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:40:02.883863 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.883830 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:40:02.884117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.884089 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" podUID="b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 21:40:02.889438 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.889413 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:40:02.917836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.917804 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:40:02.920628 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.920597 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" podUID="e3976e0e-4921-46d0-99f4-2c26733ef15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 21:40:02.922105 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.922033 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:40:02.932775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.931341 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:40:02.985625 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.983976 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:40:02.997821 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:02.995394 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:40:03.016787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:03.015525 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:40:03.016787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:03.015598 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" podUID="db7b6603-3dfd-4f20-a362-da44b2fb9da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 21:40:03.016787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:03.016492 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:40:03.105688 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:03.105494 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:03.154670 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:03.154616 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:40:03.154991 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:03.154946 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" podUID="7bdabb86-13f1-4ab6-8b08-418fff19b933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 21:40:03.862911 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:03.862875 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:04.379469 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:04.377888 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:40:04.379469 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:04.379320 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:40:04.384493 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:04.384241 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:40:04.576672 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:04.576638 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:04.978986 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:04.978914 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:40:04.979677 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:04.979642 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" podUID="b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:05.035200 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:05.035164 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:40:05.036274 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:05.036238 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:40:05.047448 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:05.047396 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:40:05.083009 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:05.082964 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:40:05.083291 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:05.083260 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" podUID="db7b6603-3dfd-4f20-a362-da44b2fb9da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:05.235175 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:05.235092 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:40:05.237674 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:05.237328 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:40:05.238947 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:05.238906 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:40:05.266595 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:05.266552 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:40:05.267853 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:05.267831 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:40:05.269275 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:05.269227 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:40:05.369129 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:05.369094 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:05.491136 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:05.491027 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:06.063700 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:06.063659 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:06.750481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:06.750441 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:07.446989 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:07.446954 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:08.243628 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.243586 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:08.320251 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.320212 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl"] Oct 30 21:40:08.326392 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.326334 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.346285 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.346249 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl"] Oct 30 21:40:08.484237 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.484206 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1615c58-e876-4519-b38d-45166e1411c5-istio-token\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.484634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.484262 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-credential-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.484634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.484356 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-istio-data\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.484634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.484399 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-workload-certs\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.484634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.484426 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1615c58-e876-4519-b38d-45166e1411c5-istio-podinfo\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.484634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.484463 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-workload-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.484634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.484509 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-istio-envoy\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.484634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.484591 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdcj9\" (UniqueName: \"kubernetes.io/projected/c1615c58-e876-4519-b38d-45166e1411c5-kube-api-access-cdcj9\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.484634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.484622 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1615c58-e876-4519-b38d-45166e1411c5-istiod-ca-cert\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.585360 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.585327 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-istio-envoy\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.585538 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.585382 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdcj9\" (UniqueName: \"kubernetes.io/projected/c1615c58-e876-4519-b38d-45166e1411c5-kube-api-access-cdcj9\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.585538 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.585409 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1615c58-e876-4519-b38d-45166e1411c5-istiod-ca-cert\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.585538 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.585506 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1615c58-e876-4519-b38d-45166e1411c5-istio-token\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.585699 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.585560 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-credential-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.585699 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.585595 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-istio-data\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.585699 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.585636 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-workload-certs\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.585699 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.585664 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1615c58-e876-4519-b38d-45166e1411c5-istio-podinfo\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.585699 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.585695 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-workload-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.585989 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.585965 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-credential-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.586131 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.586107 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-istio-data\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.586198 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.586150 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-workload-certs\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.586249 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.586198 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-workload-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.586308 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.586291 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1615c58-e876-4519-b38d-45166e1411c5-istiod-ca-cert\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.587703 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.587678 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-istio-envoy\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.588098 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.588073 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1615c58-e876-4519-b38d-45166e1411c5-istio-podinfo\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.597861 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.597823 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1615c58-e876-4519-b38d-45166e1411c5-istio-token\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.598033 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.598008 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdcj9\" (UniqueName: \"kubernetes.io/projected/c1615c58-e876-4519-b38d-45166e1411c5-kube-api-access-cdcj9\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.659961 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.659926 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:08.713585 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.713544 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89"] Oct 30 21:40:08.719445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.718385 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.751980 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.751941 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89"] Oct 30 21:40:08.889594 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.889503 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630b8106-55a0-46e3-9de9-85898b01255d-istiod-ca-cert\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.889804 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.889645 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630b8106-55a0-46e3-9de9-85898b01255d-istio-token\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.889804 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.889708 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-istio-envoy\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.889804 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.889743 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-credential-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.889947 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.889884 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-workload-certs\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.889994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.889971 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630b8106-55a0-46e3-9de9-85898b01255d-istio-podinfo\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.890033 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.889996 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-workload-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.890113 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.890088 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-istio-data\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.890149 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.890129 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9xx2\" (UniqueName: \"kubernetes.io/projected/630b8106-55a0-46e3-9de9-85898b01255d-kube-api-access-r9xx2\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.948961 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:40:08.948929 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1615c58_e876_4519_b38d_45166e1411c5.slice/crio-8e408ddf07e93287f733fbfd00b27f0e433d318da7d09a7c9f7d464c7c591d46 WatchSource:0}: Error finding container 8e408ddf07e93287f733fbfd00b27f0e433d318da7d09a7c9f7d464c7c591d46: Status 404 returned error can't find the container with id 8e408ddf07e93287f733fbfd00b27f0e433d318da7d09a7c9f7d464c7c591d46 Oct 30 21:40:08.949179 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.949155 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl"] Oct 30 21:40:08.951923 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.951892 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:08.951998 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.951954 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:08.951998 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.951984 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:08.991228 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.991200 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630b8106-55a0-46e3-9de9-85898b01255d-istiod-ca-cert\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.991363 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.991289 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630b8106-55a0-46e3-9de9-85898b01255d-istio-token\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.991363 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.991331 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-istio-envoy\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.991473 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.991377 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-credential-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.991473 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.991420 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-workload-certs\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.991577 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.991494 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630b8106-55a0-46e3-9de9-85898b01255d-istio-podinfo\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.991577 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.991528 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-workload-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.991672 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.991592 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-istio-data\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.991672 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.991619 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9xx2\" (UniqueName: \"kubernetes.io/projected/630b8106-55a0-46e3-9de9-85898b01255d-kube-api-access-r9xx2\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.991908 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.991883 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-workload-certs\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.992017 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.991948 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-credential-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.992017 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.991969 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-workload-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.992017 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.992013 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630b8106-55a0-46e3-9de9-85898b01255d-istiod-ca-cert\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.992187 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.992143 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-istio-data\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.994147 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.994119 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630b8106-55a0-46e3-9de9-85898b01255d-istio-podinfo\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:08.994244 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:08.994196 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-istio-envoy\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:09.007320 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:09.007282 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630b8106-55a0-46e3-9de9-85898b01255d-istio-token\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:09.007514 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:09.007495 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9xx2\" (UniqueName: \"kubernetes.io/projected/630b8106-55a0-46e3-9de9-85898b01255d-kube-api-access-r9xx2\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:09.047245 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:09.047209 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:09.060738 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:09.060709 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:09.390608 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:09.390544 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89"] Oct 30 21:40:09.391501 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:40:09.391476 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630b8106_55a0_46e3_9de9_85898b01255d.slice/crio-917b8c735de9eca2151b9ac753a32f0788a8de6b2353bcc9bc816d4bfd7ccc73 WatchSource:0}: Error finding container 917b8c735de9eca2151b9ac753a32f0788a8de6b2353bcc9bc816d4bfd7ccc73: Status 404 returned error can't find the container with id 917b8c735de9eca2151b9ac753a32f0788a8de6b2353bcc9bc816d4bfd7ccc73 Oct 30 21:40:09.394122 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:09.394090 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:09.394231 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:09.394167 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:09.394231 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:09.394215 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:09.484658 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:09.484612 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" event={"ID":"c1615c58-e876-4519-b38d-45166e1411c5","Type":"ContainerStarted","Data":"1fc16a853628620bf0d44a99bb9e2542a6e7e9430032942ad14e01f2e34c9da3"} Oct 30 21:40:09.485196 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:09.484683 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" event={"ID":"c1615c58-e876-4519-b38d-45166e1411c5","Type":"ContainerStarted","Data":"8e408ddf07e93287f733fbfd00b27f0e433d318da7d09a7c9f7d464c7c591d46"} Oct 30 21:40:09.489606 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:09.488737 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" event={"ID":"630b8106-55a0-46e3-9de9-85898b01255d","Type":"ContainerStarted","Data":"17f889ac80e2685467e87c968ee179f97b0ab247d9498a53dce650ea89780c92"} Oct 30 21:40:09.489606 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:09.488821 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" event={"ID":"630b8106-55a0-46e3-9de9-85898b01255d","Type":"ContainerStarted","Data":"917b8c735de9eca2151b9ac753a32f0788a8de6b2353bcc9bc816d4bfd7ccc73"} Oct 30 21:40:09.518783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:09.518691 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" podStartSLOduration=1.5186761739999999 podStartE2EDuration="1.518676174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:09.518357444 +0000 UTC m=+678.114174102" watchObservedRunningTime="2026-10-30 21:40:09.518676174 +0000 UTC m=+678.114492833" Oct 30 21:40:09.545047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:09.544992 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" podStartSLOduration=1.54497397 podStartE2EDuration="1.54497397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:09.543046062 +0000 UTC m=+678.138862723" watchObservedRunningTime="2026-10-30 21:40:09.54497397 +0000 UTC m=+678.140790632" Oct 30 21:40:09.660942 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:09.660845 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:09.757649 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:09.757441 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:09.913214 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:09.913122 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:10.065782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:10.064861 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:10.079821 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:10.075898 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:40:10.079821 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:10.075973 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" podUID="630b8106-55a0-46e3-9de9-85898b01255d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:40:10.540569 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:10.540537 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:10.679289 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:10.669908 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:40:10.679289 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:10.669976 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" podUID="c1615c58-e876-4519-b38d-45166e1411c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 21:40:11.091778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:11.088640 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:40:11.091778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:11.088698 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" podUID="630b8106-55a0-46e3-9de9-85898b01255d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:40:11.348152 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:11.348063 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:11.664839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:11.661173 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:40:11.664839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:11.661237 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" podUID="c1615c58-e876-4519-b38d-45166e1411c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 21:40:12.072304 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:12.072267 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:40:12.072599 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:12.072570 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" podUID="630b8106-55a0-46e3-9de9-85898b01255d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:40:12.119171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:12.119142 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:12.661160 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:12.661123 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:40:12.661466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:12.661435 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" podUID="c1615c58-e876-4519-b38d-45166e1411c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 21:40:12.835000 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:12.834923 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:13.072779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:13.069883 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:40:13.072779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:13.069947 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" podUID="630b8106-55a0-46e3-9de9-85898b01255d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:40:13.670778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:13.666794 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:40:13.670778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:13.666872 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" podUID="c1615c58-e876-4519-b38d-45166e1411c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 21:40:13.725789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:13.725744 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:14.061777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:14.061334 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:40:14.061777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:14.061482 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" podUID="630b8106-55a0-46e3-9de9-85898b01255d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:40:14.444259 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:14.444175 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:14.664214 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:14.664171 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:40:14.664480 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:14.664452 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" podUID="c1615c58-e876-4519-b38d-45166e1411c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 21:40:15.088044 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:15.086172 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:40:15.088044 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:15.086235 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" podUID="630b8106-55a0-46e3-9de9-85898b01255d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:40:15.126059 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:15.126025 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:15.250855 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:15.247279 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:15.660951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:15.660906 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:40:15.661135 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:15.660976 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" podUID="c1615c58-e876-4519-b38d-45166e1411c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 21:40:15.956575 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:15.956450 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:16.297482 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:16.297396 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:16.297976 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:16.297957 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:16.298776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:16.298725 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:40:16.664809 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:16.664772 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:16.665195 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:16.665169 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:16.665907 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:16.665889 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:40:16.788945 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:16.788911 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:17.759090 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:17.759052 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:18.655778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:18.655071 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:19.461407 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:19.461370 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:20.417671 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:20.417636 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:21.260445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:21.260410 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:22.041493 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:22.041458 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:22.738875 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:22.738844 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:23.038265 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.038177 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-548bdc16-t2286"] Oct 30 21:40:23.043768 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.043730 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-548bdc16-t2286" Oct 30 21:40:23.048582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.048490 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:40:23.048582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.048511 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:40:23.048582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.048541 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:40:23.048582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.048563 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:40:23.052555 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.052534 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-548bdc16-t2286"] Oct 30 21:40:23.195116 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.195087 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45db6891-8460-4a7f-a9ef-ed41e3ce06c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-548bdc16-t2286\" (UID: \"45db6891-8460-4a7f-a9ef-ed41e3ce06c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-548bdc16-t2286" Oct 30 21:40:23.195370 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.195349 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bp49\" (UniqueName: \"kubernetes.io/projected/45db6891-8460-4a7f-a9ef-ed41e3ce06c1-kube-api-access-9bp49\") pod \"maas-api-revoke-sub-548bdc16-t2286\" (UID: \"45db6891-8460-4a7f-a9ef-ed41e3ce06c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-548bdc16-t2286" Oct 30 21:40:23.195444 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.195403 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45db6891-8460-4a7f-a9ef-ed41e3ce06c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-548bdc16-t2286\" (UID: \"45db6891-8460-4a7f-a9ef-ed41e3ce06c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-548bdc16-t2286" Oct 30 21:40:23.296890 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.296811 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bp49\" (UniqueName: \"kubernetes.io/projected/45db6891-8460-4a7f-a9ef-ed41e3ce06c1-kube-api-access-9bp49\") pod \"maas-api-revoke-sub-548bdc16-t2286\" (UID: \"45db6891-8460-4a7f-a9ef-ed41e3ce06c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-548bdc16-t2286" Oct 30 21:40:23.296890 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.296870 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45db6891-8460-4a7f-a9ef-ed41e3ce06c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-548bdc16-t2286\" (UID: \"45db6891-8460-4a7f-a9ef-ed41e3ce06c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-548bdc16-t2286" Oct 30 21:40:23.297102 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.296924 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45db6891-8460-4a7f-a9ef-ed41e3ce06c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-548bdc16-t2286\" (UID: \"45db6891-8460-4a7f-a9ef-ed41e3ce06c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-548bdc16-t2286" Oct 30 21:40:23.297651 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.297629 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45db6891-8460-4a7f-a9ef-ed41e3ce06c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-548bdc16-t2286\" (UID: \"45db6891-8460-4a7f-a9ef-ed41e3ce06c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-548bdc16-t2286" Oct 30 21:40:23.299328 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.299301 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45db6891-8460-4a7f-a9ef-ed41e3ce06c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-548bdc16-t2286\" (UID: \"45db6891-8460-4a7f-a9ef-ed41e3ce06c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-548bdc16-t2286" Oct 30 21:40:23.311382 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.311361 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bp49\" (UniqueName: \"kubernetes.io/projected/45db6891-8460-4a7f-a9ef-ed41e3ce06c1-kube-api-access-9bp49\") pod \"maas-api-revoke-sub-548bdc16-t2286\" (UID: \"45db6891-8460-4a7f-a9ef-ed41e3ce06c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-548bdc16-t2286" Oct 30 21:40:23.360345 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.360320 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-548bdc16-t2286" Oct 30 21:40:23.532704 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.532670 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw"] Oct 30 21:40:23.538413 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.538390 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:40:23.548015 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.547913 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7dfff0\"/\"distinct-2-w6-7dfff0-kserve-self-signed-certs\"" Oct 30 21:40:23.551912 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.551889 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw"] Oct 30 21:40:23.592873 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.592840 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:23.647580 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:40:23.647547 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45db6891_8460_4a7f_a9ef_ed41e3ce06c1.slice/crio-0dd749a3ed3db6f222bb5b58b44d1464cdb009ad395c2c3d136ee89f9fac70b1 WatchSource:0}: Error finding container 0dd749a3ed3db6f222bb5b58b44d1464cdb009ad395c2c3d136ee89f9fac70b1: Status 404 returned error can't find the container with id 0dd749a3ed3db6f222bb5b58b44d1464cdb009ad395c2c3d136ee89f9fac70b1 Oct 30 21:40:23.647809 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.647784 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-548bdc16-t2286"] Oct 30 21:40:23.702377 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.702351 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-home\") pod \"distinct-2-w6-7dfff0-kserve-6f954c7857-654qw\" (UID: \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\") " pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:40:23.702472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.702404 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a810e8ce-0559-414c-a5ee-eafb7822dd6a-tls-certs\") pod \"distinct-2-w6-7dfff0-kserve-6f954c7857-654qw\" (UID: \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\") " pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:40:23.702513 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.702492 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv2z4\" (UniqueName: \"kubernetes.io/projected/a810e8ce-0559-414c-a5ee-eafb7822dd6a-kube-api-access-kv2z4\") pod \"distinct-2-w6-7dfff0-kserve-6f954c7857-654qw\" (UID: \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\") " pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:40:23.702613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.702600 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-model-cache\") pod \"distinct-2-w6-7dfff0-kserve-6f954c7857-654qw\" (UID: \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\") " pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:40:23.702796 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.702736 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-dshm\") pod \"distinct-2-w6-7dfff0-kserve-6f954c7857-654qw\" (UID: \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\") " pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:40:23.702840 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.702815 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-tmp-dir\") pod \"distinct-2-w6-7dfff0-kserve-6f954c7857-654qw\" (UID: \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\") " pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:40:23.711045 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.711022 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:23.805904 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.803884 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv2z4\" (UniqueName: \"kubernetes.io/projected/a810e8ce-0559-414c-a5ee-eafb7822dd6a-kube-api-access-kv2z4\") pod \"distinct-2-w6-7dfff0-kserve-6f954c7857-654qw\" (UID: \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\") " pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:40:23.805904 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.804052 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-model-cache\") pod \"distinct-2-w6-7dfff0-kserve-6f954c7857-654qw\" (UID: \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\") " pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:40:23.805904 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.804165 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-dshm\") pod \"distinct-2-w6-7dfff0-kserve-6f954c7857-654qw\" (UID: \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\") " pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:40:23.805904 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.804194 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-tmp-dir\") pod \"distinct-2-w6-7dfff0-kserve-6f954c7857-654qw\" (UID: \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\") " pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:40:23.805904 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.804270 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-home\") pod \"distinct-2-w6-7dfff0-kserve-6f954c7857-654qw\" (UID: \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\") " pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:40:23.805904 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.804368 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a810e8ce-0559-414c-a5ee-eafb7822dd6a-tls-certs\") pod \"distinct-2-w6-7dfff0-kserve-6f954c7857-654qw\" (UID: \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\") " pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:40:23.805904 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.805591 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-model-cache\") pod \"distinct-2-w6-7dfff0-kserve-6f954c7857-654qw\" (UID: \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\") " pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:40:23.805904 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.805600 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-tmp-dir\") pod \"distinct-2-w6-7dfff0-kserve-6f954c7857-654qw\" (UID: \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\") " pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:40:23.805904 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.805868 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-home\") pod \"distinct-2-w6-7dfff0-kserve-6f954c7857-654qw\" (UID: \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\") " pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:40:23.808284 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.808248 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-dshm\") pod \"distinct-2-w6-7dfff0-kserve-6f954c7857-654qw\" (UID: \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\") " pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:40:23.808741 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.808651 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a810e8ce-0559-414c-a5ee-eafb7822dd6a-tls-certs\") pod \"distinct-2-w6-7dfff0-kserve-6f954c7857-654qw\" (UID: \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\") " pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:40:23.830432 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.830393 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv2z4\" (UniqueName: \"kubernetes.io/projected/a810e8ce-0559-414c-a5ee-eafb7822dd6a-kube-api-access-kv2z4\") pod \"distinct-2-w6-7dfff0-kserve-6f954c7857-654qw\" (UID: \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\") " pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:40:23.853935 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:23.853903 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:40:24.146713 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:24.146682 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw"] Oct 30 21:40:24.147128 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:40:24.147106 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda810e8ce_0559_414c_a5ee_eafb7822dd6a.slice/crio-65d9a05f47909c22d6802edd99c9e65550c585348fbedfd42a39db3afd183729 WatchSource:0}: Error finding container 65d9a05f47909c22d6802edd99c9e65550c585348fbedfd42a39db3afd183729: Status 404 returned error can't find the container with id 65d9a05f47909c22d6802edd99c9e65550c585348fbedfd42a39db3afd183729 Oct 30 21:40:24.639265 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:24.639229 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" event={"ID":"a810e8ce-0559-414c-a5ee-eafb7822dd6a","Type":"ContainerStarted","Data":"f8a0946cab502355d7673a363fc17963df2aac28f3a8806e2607ad696c6161c0"} Oct 30 21:40:24.639407 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:24.639282 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" event={"ID":"a810e8ce-0559-414c-a5ee-eafb7822dd6a","Type":"ContainerStarted","Data":"65d9a05f47909c22d6802edd99c9e65550c585348fbedfd42a39db3afd183729"} Oct 30 21:40:24.640745 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:24.640722 5801 generic.go:356] "Generic (PLEG): container finished" podID="45db6891-8460-4a7f-a9ef-ed41e3ce06c1" containerID="c88b73d1245a102a6039862596f35bbdbe645a26b2137eac667837cd8ffc4012" exitCode=0 Oct 30 21:40:24.640874 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:24.640778 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-548bdc16-t2286" event={"ID":"45db6891-8460-4a7f-a9ef-ed41e3ce06c1","Type":"ContainerDied","Data":"c88b73d1245a102a6039862596f35bbdbe645a26b2137eac667837cd8ffc4012"} Oct 30 21:40:24.640874 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:24.640829 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-548bdc16-t2286" event={"ID":"45db6891-8460-4a7f-a9ef-ed41e3ce06c1","Type":"ContainerStarted","Data":"0dd749a3ed3db6f222bb5b58b44d1464cdb009ad395c2c3d136ee89f9fac70b1"} Oct 30 21:40:24.649066 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:24.649044 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:24.661473 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:24.661435 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podStartSLOduration=1.661424899 podStartE2EDuration="1.661424899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:24.658232036 +0000 UTC m=+693.254048725" watchObservedRunningTime="2026-10-30 21:40:24.661424899 +0000 UTC m=+693.257241558" Oct 30 21:40:24.854895 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:24.854818 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:40:25.320323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:25.320293 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2828ee03-n9ncr"] Oct 30 21:40:25.324021 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:25.324003 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2828ee03-n9ncr" Oct 30 21:40:25.348138 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:25.338216 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2828ee03-n9ncr"] Oct 30 21:40:25.430241 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:25.430205 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a80fd8-1d28-42c0-ba24-7a3fb3db1886-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2828ee03-n9ncr\" (UID: \"34a80fd8-1d28-42c0-ba24-7a3fb3db1886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2828ee03-n9ncr" Oct 30 21:40:25.430396 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:25.430330 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dx6k\" (UniqueName: \"kubernetes.io/projected/34a80fd8-1d28-42c0-ba24-7a3fb3db1886-kube-api-access-9dx6k\") pod \"maas-api-revoke-sub-2828ee03-n9ncr\" (UID: \"34a80fd8-1d28-42c0-ba24-7a3fb3db1886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2828ee03-n9ncr" Oct 30 21:40:25.430518 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:25.430505 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a80fd8-1d28-42c0-ba24-7a3fb3db1886-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2828ee03-n9ncr\" (UID: \"34a80fd8-1d28-42c0-ba24-7a3fb3db1886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2828ee03-n9ncr" Oct 30 21:40:25.531323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:25.531287 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:25.531482 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:25.531448 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a80fd8-1d28-42c0-ba24-7a3fb3db1886-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2828ee03-n9ncr\" (UID: \"34a80fd8-1d28-42c0-ba24-7a3fb3db1886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2828ee03-n9ncr" Oct 30 21:40:25.531526 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:25.531496 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a80fd8-1d28-42c0-ba24-7a3fb3db1886-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2828ee03-n9ncr\" (UID: \"34a80fd8-1d28-42c0-ba24-7a3fb3db1886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2828ee03-n9ncr" Oct 30 21:40:25.531570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:25.531552 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dx6k\" (UniqueName: \"kubernetes.io/projected/34a80fd8-1d28-42c0-ba24-7a3fb3db1886-kube-api-access-9dx6k\") pod \"maas-api-revoke-sub-2828ee03-n9ncr\" (UID: \"34a80fd8-1d28-42c0-ba24-7a3fb3db1886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2828ee03-n9ncr" Oct 30 21:40:25.532229 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:25.532209 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a80fd8-1d28-42c0-ba24-7a3fb3db1886-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2828ee03-n9ncr\" (UID: \"34a80fd8-1d28-42c0-ba24-7a3fb3db1886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2828ee03-n9ncr" Oct 30 21:40:25.533904 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:25.533881 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a80fd8-1d28-42c0-ba24-7a3fb3db1886-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2828ee03-n9ncr\" (UID: \"34a80fd8-1d28-42c0-ba24-7a3fb3db1886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2828ee03-n9ncr" Oct 30 21:40:25.544073 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:25.544049 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dx6k\" (UniqueName: \"kubernetes.io/projected/34a80fd8-1d28-42c0-ba24-7a3fb3db1886-kube-api-access-9dx6k\") pod \"maas-api-revoke-sub-2828ee03-n9ncr\" (UID: \"34a80fd8-1d28-42c0-ba24-7a3fb3db1886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2828ee03-n9ncr" Oct 30 21:40:25.652528 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:25.652499 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2828ee03-n9ncr" Oct 30 21:40:25.831711 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:25.831688 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-548bdc16-t2286" Oct 30 21:40:25.937023 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:25.936953 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45db6891-8460-4a7f-a9ef-ed41e3ce06c1-kube-api-access-9bp49\" (UniqueName: \"kubernetes.io/projected/45db6891-8460-4a7f-a9ef-ed41e3ce06c1-kube-api-access-9bp49\") pod \"45db6891-8460-4a7f-a9ef-ed41e3ce06c1\" (UID: \"45db6891-8460-4a7f-a9ef-ed41e3ce06c1\") " Oct 30 21:40:25.937424 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:25.937124 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45db6891-8460-4a7f-a9ef-ed41e3ce06c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45db6891-8460-4a7f-a9ef-ed41e3ce06c1-maas-api-xks-ca\") pod \"45db6891-8460-4a7f-a9ef-ed41e3ce06c1\" (UID: \"45db6891-8460-4a7f-a9ef-ed41e3ce06c1\") " Oct 30 21:40:25.937424 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:25.937170 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45db6891-8460-4a7f-a9ef-ed41e3ce06c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45db6891-8460-4a7f-a9ef-ed41e3ce06c1-maas-api-openshift-ca\") pod \"45db6891-8460-4a7f-a9ef-ed41e3ce06c1\" (UID: \"45db6891-8460-4a7f-a9ef-ed41e3ce06c1\") " Oct 30 21:40:25.937609 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:25.937587 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45db6891-8460-4a7f-a9ef-ed41e3ce06c1-maas-api-openshift-ca" pod "45db6891-8460-4a7f-a9ef-ed41e3ce06c1" (UID: "45db6891-8460-4a7f-a9ef-ed41e3ce06c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:25.937729 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:25.937713 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45db6891-8460-4a7f-a9ef-ed41e3ce06c1-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:25.938984 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:25.938964 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45db6891-8460-4a7f-a9ef-ed41e3ce06c1-kube-api-access-9bp49" pod "45db6891-8460-4a7f-a9ef-ed41e3ce06c1" (UID: "45db6891-8460-4a7f-a9ef-ed41e3ce06c1"). InnerVolumeSpecName "kube-api-access-9bp49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:25.939492 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:25.939472 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45db6891-8460-4a7f-a9ef-ed41e3ce06c1-maas-api-xks-ca" pod "45db6891-8460-4a7f-a9ef-ed41e3ce06c1" (UID: "45db6891-8460-4a7f-a9ef-ed41e3ce06c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:25.947618 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:40:25.947595 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a80fd8_1d28_42c0_ba24_7a3fb3db1886.slice/crio-9d04d43dba5dadb824f0cc648b66ed10b96cbda6033a3778eaf8d9bd9085d704 WatchSource:0}: Error finding container 9d04d43dba5dadb824f0cc648b66ed10b96cbda6033a3778eaf8d9bd9085d704: Status 404 returned error can't find the container with id 9d04d43dba5dadb824f0cc648b66ed10b96cbda6033a3778eaf8d9bd9085d704 Oct 30 21:40:25.948213 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:25.948192 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2828ee03-n9ncr"] Oct 30 21:40:26.039389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:26.039360 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45db6891-8460-4a7f-a9ef-ed41e3ce06c1-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:26.039389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:26.039387 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bp49\" (UniqueName: \"kubernetes.io/projected/45db6891-8460-4a7f-a9ef-ed41e3ce06c1-kube-api-access-9bp49\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:26.351406 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:26.351368 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:26.657092 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:26.657048 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2828ee03-n9ncr" event={"ID":"34a80fd8-1d28-42c0-ba24-7a3fb3db1886","Type":"ContainerStarted","Data":"bd661adbe59eefe3265d1abd39b46fd24b64df2cddb3653812eb004413d06b7b"} Oct 30 21:40:26.657289 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:26.657116 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2828ee03-n9ncr" event={"ID":"34a80fd8-1d28-42c0-ba24-7a3fb3db1886","Type":"ContainerStarted","Data":"9d04d43dba5dadb824f0cc648b66ed10b96cbda6033a3778eaf8d9bd9085d704"} Oct 30 21:40:26.658615 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:26.658591 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-548bdc16-t2286" Oct 30 21:40:26.658615 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:26.658596 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-548bdc16-t2286" event={"ID":"45db6891-8460-4a7f-a9ef-ed41e3ce06c1","Type":"ContainerDied","Data":"0dd749a3ed3db6f222bb5b58b44d1464cdb009ad395c2c3d136ee89f9fac70b1"} Oct 30 21:40:26.658776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:26.658642 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dd749a3ed3db6f222bb5b58b44d1464cdb009ad395c2c3d136ee89f9fac70b1" Oct 30 21:40:26.680858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:26.680816 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2828ee03-n9ncr" podStartSLOduration=1.680803741 podStartE2EDuration="1.680803741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:26.676597451 +0000 UTC m=+695.272414111" watchObservedRunningTime="2026-10-30 21:40:26.680803741 +0000 UTC m=+695.276620401" Oct 30 21:40:27.125423 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:27.125343 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:27.667243 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:27.667206 5801 generic.go:356] "Generic (PLEG): container finished" podID="34a80fd8-1d28-42c0-ba24-7a3fb3db1886" containerID="bd661adbe59eefe3265d1abd39b46fd24b64df2cddb3653812eb004413d06b7b" exitCode=0 Oct 30 21:40:27.667409 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:27.667286 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2828ee03-n9ncr" event={"ID":"34a80fd8-1d28-42c0-ba24-7a3fb3db1886","Type":"ContainerDied","Data":"bd661adbe59eefe3265d1abd39b46fd24b64df2cddb3653812eb004413d06b7b"} Oct 30 21:40:28.011100 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:28.011016 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:28.725231 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:28.725198 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:28.834741 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:28.834715 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2828ee03-n9ncr" Oct 30 21:40:28.878502 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:28.878467 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34a80fd8-1d28-42c0-ba24-7a3fb3db1886-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a80fd8-1d28-42c0-ba24-7a3fb3db1886-maas-api-openshift-ca\") pod \"34a80fd8-1d28-42c0-ba24-7a3fb3db1886\" (UID: \"34a80fd8-1d28-42c0-ba24-7a3fb3db1886\") " Oct 30 21:40:28.878691 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:28.878598 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a80fd8-1d28-42c0-ba24-7a3fb3db1886-kube-api-access-9dx6k\" (UniqueName: \"kubernetes.io/projected/34a80fd8-1d28-42c0-ba24-7a3fb3db1886-kube-api-access-9dx6k\") pod \"34a80fd8-1d28-42c0-ba24-7a3fb3db1886\" (UID: \"34a80fd8-1d28-42c0-ba24-7a3fb3db1886\") " Oct 30 21:40:28.878691 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:28.878673 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34a80fd8-1d28-42c0-ba24-7a3fb3db1886-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a80fd8-1d28-42c0-ba24-7a3fb3db1886-maas-api-xks-ca\") pod \"34a80fd8-1d28-42c0-ba24-7a3fb3db1886\" (UID: \"34a80fd8-1d28-42c0-ba24-7a3fb3db1886\") " Oct 30 21:40:28.878891 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:28.878860 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a80fd8-1d28-42c0-ba24-7a3fb3db1886-maas-api-openshift-ca" pod "34a80fd8-1d28-42c0-ba24-7a3fb3db1886" (UID: "34a80fd8-1d28-42c0-ba24-7a3fb3db1886"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:28.879143 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:28.879124 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a80fd8-1d28-42c0-ba24-7a3fb3db1886-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:28.880739 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:28.880709 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a80fd8-1d28-42c0-ba24-7a3fb3db1886-kube-api-access-9dx6k" pod "34a80fd8-1d28-42c0-ba24-7a3fb3db1886" (UID: "34a80fd8-1d28-42c0-ba24-7a3fb3db1886"). InnerVolumeSpecName "kube-api-access-9dx6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:28.881213 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:28.881192 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34a80fd8-1d28-42c0-ba24-7a3fb3db1886-maas-api-xks-ca" pod "34a80fd8-1d28-42c0-ba24-7a3fb3db1886" (UID: "34a80fd8-1d28-42c0-ba24-7a3fb3db1886"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:28.979991 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:28.979930 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dx6k\" (UniqueName: \"kubernetes.io/projected/34a80fd8-1d28-42c0-ba24-7a3fb3db1886-kube-api-access-9dx6k\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:28.979991 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:28.979955 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a80fd8-1d28-42c0-ba24-7a3fb3db1886-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:29.444114 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:29.444074 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:29.682886 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:29.682858 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2828ee03-n9ncr" Oct 30 21:40:29.683052 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:29.682857 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2828ee03-n9ncr" event={"ID":"34a80fd8-1d28-42c0-ba24-7a3fb3db1886","Type":"ContainerDied","Data":"9d04d43dba5dadb824f0cc648b66ed10b96cbda6033a3778eaf8d9bd9085d704"} Oct 30 21:40:29.683052 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:29.682981 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d04d43dba5dadb824f0cc648b66ed10b96cbda6033a3778eaf8d9bd9085d704" Oct 30 21:40:30.165839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:30.165808 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:30.282234 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:40:30.282175 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:40:30.483828 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:30.483723 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:31.042067 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.042032 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:31.396233 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.396195 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12519330-wfb6k"] Oct 30 21:40:31.396745 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.396730 5801 cpu_manager.go:414] "containerMap: removing container" podUID="45db6891-8460-4a7f-a9ef-ed41e3ce06c1" containerName="revoke-keys" Oct 30 21:40:31.396745 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.396746 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45db6891-8460-4a7f-a9ef-ed41e3ce06c1" containerName="revoke-keys" Oct 30 21:40:31.396845 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.396813 5801 cpu_manager.go:414] "containerMap: removing container" podUID="34a80fd8-1d28-42c0-ba24-7a3fb3db1886" containerName="revoke-keys" Oct 30 21:40:31.396845 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.396820 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a80fd8-1d28-42c0-ba24-7a3fb3db1886" containerName="revoke-keys" Oct 30 21:40:31.396917 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.396906 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a80fd8-1d28-42c0-ba24-7a3fb3db1886" containerName="revoke-keys" Oct 30 21:40:31.396959 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.396920 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="45db6891-8460-4a7f-a9ef-ed41e3ce06c1" containerName="revoke-keys" Oct 30 21:40:31.400654 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.400638 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12519330-wfb6k" Oct 30 21:40:31.405324 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.405295 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:40:31.405437 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.405350 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:40:31.405437 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.405355 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:40:31.405437 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.405394 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:40:31.410156 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.410131 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12519330-wfb6k"] Oct 30 21:40:31.510798 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.510767 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2b05c52-7674-4755-8418-96a26244f178-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12519330-wfb6k\" (UID: \"a2b05c52-7674-4755-8418-96a26244f178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12519330-wfb6k" Oct 30 21:40:31.510944 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.510814 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scbls\" (UniqueName: \"kubernetes.io/projected/a2b05c52-7674-4755-8418-96a26244f178-kube-api-access-scbls\") pod \"maas-api-revoke-sub-12519330-wfb6k\" (UID: \"a2b05c52-7674-4755-8418-96a26244f178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12519330-wfb6k" Oct 30 21:40:31.510944 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.510932 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2b05c52-7674-4755-8418-96a26244f178-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12519330-wfb6k\" (UID: \"a2b05c52-7674-4755-8418-96a26244f178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12519330-wfb6k" Oct 30 21:40:31.611908 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.611871 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbls\" (UniqueName: \"kubernetes.io/projected/a2b05c52-7674-4755-8418-96a26244f178-kube-api-access-scbls\") pod \"maas-api-revoke-sub-12519330-wfb6k\" (UID: \"a2b05c52-7674-4755-8418-96a26244f178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12519330-wfb6k" Oct 30 21:40:31.612080 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.611994 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2b05c52-7674-4755-8418-96a26244f178-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12519330-wfb6k\" (UID: \"a2b05c52-7674-4755-8418-96a26244f178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12519330-wfb6k" Oct 30 21:40:31.612279 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.612256 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2b05c52-7674-4755-8418-96a26244f178-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12519330-wfb6k\" (UID: \"a2b05c52-7674-4755-8418-96a26244f178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12519330-wfb6k" Oct 30 21:40:31.612917 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.612898 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2b05c52-7674-4755-8418-96a26244f178-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12519330-wfb6k\" (UID: \"a2b05c52-7674-4755-8418-96a26244f178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12519330-wfb6k" Oct 30 21:40:31.614451 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.614430 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2b05c52-7674-4755-8418-96a26244f178-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12519330-wfb6k\" (UID: \"a2b05c52-7674-4755-8418-96a26244f178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12519330-wfb6k" Oct 30 21:40:31.646051 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.646021 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scbls\" (UniqueName: \"kubernetes.io/projected/a2b05c52-7674-4755-8418-96a26244f178-kube-api-access-scbls\") pod \"maas-api-revoke-sub-12519330-wfb6k\" (UID: \"a2b05c52-7674-4755-8418-96a26244f178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12519330-wfb6k" Oct 30 21:40:31.715435 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.715379 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12519330-wfb6k" Oct 30 21:40:31.947233 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.947198 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:31.996178 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:40:31.996144 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b05c52_7674_4755_8418_96a26244f178.slice/crio-46218179f2554764a27ab4e7d738f896fff19c108eff55a32fcbee9c8bc23796 WatchSource:0}: Error finding container 46218179f2554764a27ab4e7d738f896fff19c108eff55a32fcbee9c8bc23796: Status 404 returned error can't find the container with id 46218179f2554764a27ab4e7d738f896fff19c108eff55a32fcbee9c8bc23796 Oct 30 21:40:31.996839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:31.996814 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12519330-wfb6k"] Oct 30 21:40:32.658359 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:32.658318 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:32.705943 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:32.705911 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12519330-wfb6k" event={"ID":"a2b05c52-7674-4755-8418-96a26244f178","Type":"ContainerStarted","Data":"cfae2044b1b0edd0ab1302891e7532f8abe2f1abca54279b728e4654a99146e3"} Oct 30 21:40:32.706127 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:32.706009 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12519330-wfb6k" event={"ID":"a2b05c52-7674-4755-8418-96a26244f178","Type":"ContainerStarted","Data":"46218179f2554764a27ab4e7d738f896fff19c108eff55a32fcbee9c8bc23796"} Oct 30 21:40:32.734785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:32.734715 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12519330-wfb6k" podStartSLOduration=1.734700507 podStartE2EDuration="1.734700507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:32.72981899 +0000 UTC m=+701.325635651" watchObservedRunningTime="2026-10-30 21:40:32.734700507 +0000 UTC m=+701.330517168" Oct 30 21:40:33.545054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:33.545008 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:33.717377 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:33.717343 5801 generic.go:356] "Generic (PLEG): container finished" podID="a2b05c52-7674-4755-8418-96a26244f178" containerID="cfae2044b1b0edd0ab1302891e7532f8abe2f1abca54279b728e4654a99146e3" exitCode=0 Oct 30 21:40:33.717903 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:33.717399 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12519330-wfb6k" event={"ID":"a2b05c52-7674-4755-8418-96a26244f178","Type":"ContainerDied","Data":"cfae2044b1b0edd0ab1302891e7532f8abe2f1abca54279b728e4654a99146e3"} Oct 30 21:40:33.854360 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:33.854325 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:40:33.879775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:33.878345 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:40:34.526777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:34.526659 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:35.497797 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:35.496139 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:40:35.517192 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:35.514839 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:40:35.517192 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:35.514905 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" podUID="e3976e0e-4921-46d0-99f4-2c26733ef15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:35.517192 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:35.515089 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:35.517192 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:35.515386 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:40:35.555715 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:35.555400 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:35.565529 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:35.565492 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:35.843774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:35.841573 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12519330-wfb6k" Oct 30 21:40:35.926850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:35.910938 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.067059 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:36.066686 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12519330-wfb6k" Oct 30 21:40:36.074498 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:36.067862 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12519330-wfb6k" event={"ID":"a2b05c52-7674-4755-8418-96a26244f178","Type":"ContainerDied","Data":"46218179f2554764a27ab4e7d738f896fff19c108eff55a32fcbee9c8bc23796"} Oct 30 21:40:36.074776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:36.074740 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46218179f2554764a27ab4e7d738f896fff19c108eff55a32fcbee9c8bc23796" Oct 30 21:40:36.074916 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:36.070841 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:40:36.150918 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:36.149965 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2b05c52-7674-4755-8418-96a26244f178-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2b05c52-7674-4755-8418-96a26244f178-maas-api-openshift-ca\") pod \"a2b05c52-7674-4755-8418-96a26244f178\" (UID: \"a2b05c52-7674-4755-8418-96a26244f178\") " Oct 30 21:40:36.150918 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:36.150090 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2b05c52-7674-4755-8418-96a26244f178-kube-api-access-scbls\" (UniqueName: \"kubernetes.io/projected/a2b05c52-7674-4755-8418-96a26244f178-kube-api-access-scbls\") pod \"a2b05c52-7674-4755-8418-96a26244f178\" (UID: \"a2b05c52-7674-4755-8418-96a26244f178\") " Oct 30 21:40:36.150918 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:36.150190 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2b05c52-7674-4755-8418-96a26244f178-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2b05c52-7674-4755-8418-96a26244f178-maas-api-xks-ca\") pod \"a2b05c52-7674-4755-8418-96a26244f178\" (UID: \"a2b05c52-7674-4755-8418-96a26244f178\") " Oct 30 21:40:36.154247 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:36.154216 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2b05c52-7674-4755-8418-96a26244f178-maas-api-openshift-ca" pod "a2b05c52-7674-4755-8418-96a26244f178" (UID: "a2b05c52-7674-4755-8418-96a26244f178"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:36.210777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:36.187718 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b05c52-7674-4755-8418-96a26244f178-kube-api-access-scbls" pod "a2b05c52-7674-4755-8418-96a26244f178" (UID: "a2b05c52-7674-4755-8418-96a26244f178"). InnerVolumeSpecName "kube-api-access-scbls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:36.212847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:36.211177 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2b05c52-7674-4755-8418-96a26244f178-maas-api-xks-ca" pod "a2b05c52-7674-4755-8418-96a26244f178" (UID: "a2b05c52-7674-4755-8418-96a26244f178"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:36.251941 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:36.251908 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scbls\" (UniqueName: \"kubernetes.io/projected/a2b05c52-7674-4755-8418-96a26244f178-kube-api-access-scbls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:36.251941 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:36.251936 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2b05c52-7674-4755-8418-96a26244f178-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:36.251941 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:36.251946 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2b05c52-7674-4755-8418-96a26244f178-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:36.553913 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:36.553883 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:37.233074 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.233043 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298"] Oct 30 21:40:37.233612 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.233597 5801 cpu_manager.go:414] "containerMap: removing container" podUID="a2b05c52-7674-4755-8418-96a26244f178" containerName="revoke-keys" Oct 30 21:40:37.233675 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.233614 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2b05c52-7674-4755-8418-96a26244f178" containerName="revoke-keys" Oct 30 21:40:37.233724 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.233713 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2b05c52-7674-4755-8418-96a26244f178" containerName="revoke-keys" Oct 30 21:40:37.237212 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.237192 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:40:37.246053 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.246029 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-db0b54\"/\"kube-root-ca.crt\"" Oct 30 21:40:37.247168 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.247145 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-db0b54\"/\"default-dockercfg-ldhl4\"" Oct 30 21:40:37.247316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.247238 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-db0b54\"/\"distinct-2-w2-db0b54-kserve-self-signed-certs\"" Oct 30 21:40:37.247316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.247238 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-db0b54\"/\"openshift-service-ca.crt\"" Oct 30 21:40:37.248845 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.248825 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298"] Oct 30 21:40:37.259929 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.259899 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:37.365947 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.365905 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-dshm\") pod \"distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298\" (UID: \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:40:37.366142 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.365975 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-model-cache\") pod \"distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298\" (UID: \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:40:37.366191 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.366158 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btdbc\" (UniqueName: \"kubernetes.io/projected/e33e49d7-3ed1-4ec8-ab79-de53674ea214-kube-api-access-btdbc\") pod \"distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298\" (UID: \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:40:37.366233 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.366203 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33e49d7-3ed1-4ec8-ab79-de53674ea214-tls-certs\") pod \"distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298\" (UID: \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:40:37.366329 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.366313 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-tmp-dir\") pod \"distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298\" (UID: \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:40:37.366438 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.366421 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-home\") pod \"distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298\" (UID: \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:40:37.468068 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.468034 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-home\") pod \"distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298\" (UID: \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:40:37.468267 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.468109 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-dshm\") pod \"distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298\" (UID: \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:40:37.468267 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.468143 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-model-cache\") pod \"distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298\" (UID: \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:40:37.468267 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.468230 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btdbc\" (UniqueName: \"kubernetes.io/projected/e33e49d7-3ed1-4ec8-ab79-de53674ea214-kube-api-access-btdbc\") pod \"distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298\" (UID: \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:40:37.468267 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.468260 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33e49d7-3ed1-4ec8-ab79-de53674ea214-tls-certs\") pod \"distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298\" (UID: \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:40:37.468503 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.468325 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-tmp-dir\") pod \"distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298\" (UID: \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:40:37.468503 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.468488 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-home\") pod \"distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298\" (UID: \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:40:37.468710 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.468660 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-model-cache\") pod \"distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298\" (UID: \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:40:37.468710 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.468698 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-tmp-dir\") pod \"distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298\" (UID: \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:40:37.470501 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.470479 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-dshm\") pod \"distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298\" (UID: \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:40:37.470913 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.470894 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33e49d7-3ed1-4ec8-ab79-de53674ea214-tls-certs\") pod \"distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298\" (UID: \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:40:37.489157 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.489067 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btdbc\" (UniqueName: \"kubernetes.io/projected/e33e49d7-3ed1-4ec8-ab79-de53674ea214-kube-api-access-btdbc\") pod \"distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298\" (UID: \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\") " pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:40:37.554638 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.554608 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:40:37.671649 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.671617 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:37.839974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.839943 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8eb4f19-7455v"] Oct 30 21:40:37.844999 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.844981 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8eb4f19-7455v" Oct 30 21:40:37.851486 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.851462 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:40:37.851635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.851620 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:40:37.851772 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.851733 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:40:37.852328 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.852309 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:40:37.854736 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.854716 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8eb4f19-7455v"] Oct 30 21:40:37.860035 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.860015 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298"] Oct 30 21:40:37.861881 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:40:37.861858 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode33e49d7_3ed1_4ec8_ab79_de53674ea214.slice/crio-e3f0d8ce003ba1df04f96efde0d854c606a8e6e58c458c1fb7e4989434b57072 WatchSource:0}: Error finding container e3f0d8ce003ba1df04f96efde0d854c606a8e6e58c458c1fb7e4989434b57072: Status 404 returned error can't find the container with id e3f0d8ce003ba1df04f96efde0d854c606a8e6e58c458c1fb7e4989434b57072 Oct 30 21:40:37.975221 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.975186 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgzmp\" (UniqueName: \"kubernetes.io/projected/b0509334-ceac-4802-8bcf-780c4cf295de-kube-api-access-lgzmp\") pod \"maas-api-revoke-sub-f8eb4f19-7455v\" (UID: \"b0509334-ceac-4802-8bcf-780c4cf295de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8eb4f19-7455v" Oct 30 21:40:37.975343 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.975258 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0509334-ceac-4802-8bcf-780c4cf295de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8eb4f19-7455v\" (UID: \"b0509334-ceac-4802-8bcf-780c4cf295de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8eb4f19-7455v" Oct 30 21:40:37.975445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:37.975424 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0509334-ceac-4802-8bcf-780c4cf295de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8eb4f19-7455v\" (UID: \"b0509334-ceac-4802-8bcf-780c4cf295de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8eb4f19-7455v" Oct 30 21:40:38.076792 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:38.076765 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0509334-ceac-4802-8bcf-780c4cf295de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8eb4f19-7455v\" (UID: \"b0509334-ceac-4802-8bcf-780c4cf295de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8eb4f19-7455v" Oct 30 21:40:38.076928 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:38.076910 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgzmp\" (UniqueName: \"kubernetes.io/projected/b0509334-ceac-4802-8bcf-780c4cf295de-kube-api-access-lgzmp\") pod \"maas-api-revoke-sub-f8eb4f19-7455v\" (UID: \"b0509334-ceac-4802-8bcf-780c4cf295de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8eb4f19-7455v" Oct 30 21:40:38.076980 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:38.076948 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0509334-ceac-4802-8bcf-780c4cf295de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8eb4f19-7455v\" (UID: \"b0509334-ceac-4802-8bcf-780c4cf295de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8eb4f19-7455v" Oct 30 21:40:38.077478 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:38.077452 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0509334-ceac-4802-8bcf-780c4cf295de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8eb4f19-7455v\" (UID: \"b0509334-ceac-4802-8bcf-780c4cf295de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8eb4f19-7455v" Oct 30 21:40:38.079237 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:38.079216 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0509334-ceac-4802-8bcf-780c4cf295de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8eb4f19-7455v\" (UID: \"b0509334-ceac-4802-8bcf-780c4cf295de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8eb4f19-7455v" Oct 30 21:40:38.082062 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:38.082041 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" event={"ID":"e33e49d7-3ed1-4ec8-ab79-de53674ea214","Type":"ContainerStarted","Data":"2e2aff92c8b972908623a2676ced1c83afdc70f56a69d179fefbe42c90b4464a"} Oct 30 21:40:38.082148 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:38.082076 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" event={"ID":"e33e49d7-3ed1-4ec8-ab79-de53674ea214","Type":"ContainerStarted","Data":"e3f0d8ce003ba1df04f96efde0d854c606a8e6e58c458c1fb7e4989434b57072"} Oct 30 21:40:38.091567 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:38.091518 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgzmp\" (UniqueName: \"kubernetes.io/projected/b0509334-ceac-4802-8bcf-780c4cf295de-kube-api-access-lgzmp\") pod \"maas-api-revoke-sub-f8eb4f19-7455v\" (UID: \"b0509334-ceac-4802-8bcf-780c4cf295de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8eb4f19-7455v" Oct 30 21:40:38.102999 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:38.102961 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podStartSLOduration=1.102950412 podStartE2EDuration="1.102950412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:38.100518693 +0000 UTC m=+706.696335365" watchObservedRunningTime="2026-10-30 21:40:38.102950412 +0000 UTC m=+706.698767071" Oct 30 21:40:38.161816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:38.161785 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8eb4f19-7455v" Oct 30 21:40:38.440831 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:40:38.440785 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0509334_ceac_4802_8bcf_780c4cf295de.slice/crio-5a9f6c11c6aef0014258e6770b5d8f9da2bf554b80d5f8032ea01b029b07f54b WatchSource:0}: Error finding container 5a9f6c11c6aef0014258e6770b5d8f9da2bf554b80d5f8032ea01b029b07f54b: Status 404 returned error can't find the container with id 5a9f6c11c6aef0014258e6770b5d8f9da2bf554b80d5f8032ea01b029b07f54b Oct 30 21:40:38.441281 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:38.441257 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8eb4f19-7455v"] Oct 30 21:40:38.454272 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:38.454239 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:38.555922 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:38.555878 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:40:39.078230 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.078197 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h"] Oct 30 21:40:39.082127 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.082109 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.091305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.091279 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8eb4f19-7455v" event={"ID":"b0509334-ceac-4802-8bcf-780c4cf295de","Type":"ContainerStarted","Data":"f89d8ed1a025e9a37f57c52692c407e21dcd73c1d5e49e4347e6dcd4744e8368"} Oct 30 21:40:39.091428 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.091320 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8eb4f19-7455v" event={"ID":"b0509334-ceac-4802-8bcf-780c4cf295de","Type":"ContainerStarted","Data":"5a9f6c11c6aef0014258e6770b5d8f9da2bf554b80d5f8032ea01b029b07f54b"} Oct 30 21:40:39.098490 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.098468 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h"] Oct 30 21:40:39.141055 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.140969 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8eb4f19-7455v" podStartSLOduration=2.1409566939999998 podStartE2EDuration="2.140956694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:39.139143167 +0000 UTC m=+707.734959840" watchObservedRunningTime="2026-10-30 21:40:39.140956694 +0000 UTC m=+707.736773353" Oct 30 21:40:39.190943 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.190914 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-workload-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.191078 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.190984 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-envoy\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.191078 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.191002 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-podinfo\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.191157 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.191106 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd2st\" (UniqueName: \"kubernetes.io/projected/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-kube-api-access-vd2st\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.191238 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.191214 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-token\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.191308 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.191280 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istiod-ca-cert\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.191435 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.191350 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-workload-certs\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.191539 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.191518 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-credential-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.191599 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.191557 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-data\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.252992 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.252961 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:39.292544 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.292516 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-workload-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.292668 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.292567 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-envoy\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.292668 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.292596 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-podinfo\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.292800 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.292671 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd2st\" (UniqueName: \"kubernetes.io/projected/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-kube-api-access-vd2st\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.292800 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.292773 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-token\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.292905 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.292811 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istiod-ca-cert\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.292905 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.292896 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-workload-certs\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.293009 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.292955 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-workload-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.293238 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.293207 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-credential-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.293343 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.293266 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-data\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.293436 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.293394 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-workload-certs\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.293643 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.293613 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-data\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.293778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.293739 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istiod-ca-cert\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.293933 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.293913 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-credential-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.295148 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.295123 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-envoy\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.295595 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.295573 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-podinfo\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.306386 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.306352 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-token\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.307741 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.307721 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd2st\" (UniqueName: \"kubernetes.io/projected/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-kube-api-access-vd2st\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.397996 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.397912 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:39.511576 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:40:39.511536 5801 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0509334_ceac_4802_8bcf_780c4cf295de.slice/crio-f89d8ed1a025e9a37f57c52692c407e21dcd73c1d5e49e4347e6dcd4744e8368.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0509334_ceac_4802_8bcf_780c4cf295de.slice/crio-conmon-f89d8ed1a025e9a37f57c52692c407e21dcd73c1d5e49e4347e6dcd4744e8368.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:40:39.687375 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:40:39.687342 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f59fa92_8480_42ba_b9e7_424c3b05b8e2.slice/crio-84895d7861824fc870bfeaedce7b4adde85c9646116774d1b317cb8dad0177e2 WatchSource:0}: Error finding container 84895d7861824fc870bfeaedce7b4adde85c9646116774d1b317cb8dad0177e2: Status 404 returned error can't find the container with id 84895d7861824fc870bfeaedce7b4adde85c9646116774d1b317cb8dad0177e2 Oct 30 21:40:39.687962 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.687938 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h"] Oct 30 21:40:39.690043 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.690014 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:39.690121 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.690072 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:39.690121 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:39.690102 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:40.081812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:40.081748 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:40.108730 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:40.108693 5801 generic.go:356] "Generic (PLEG): container finished" podID="b0509334-ceac-4802-8bcf-780c4cf295de" containerID="f89d8ed1a025e9a37f57c52692c407e21dcd73c1d5e49e4347e6dcd4744e8368" exitCode=0 Oct 30 21:40:40.108963 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:40.108800 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8eb4f19-7455v" event={"ID":"b0509334-ceac-4802-8bcf-780c4cf295de","Type":"ContainerDied","Data":"f89d8ed1a025e9a37f57c52692c407e21dcd73c1d5e49e4347e6dcd4744e8368"} Oct 30 21:40:40.113008 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:40.112970 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" event={"ID":"0f59fa92-8480-42ba-b9e7-424c3b05b8e2","Type":"ContainerStarted","Data":"1d7ae2f796f1490b0d04c2872a1b5c2c742673b4bba79935541de8dd02c164b1"} Oct 30 21:40:40.113276 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:40.113251 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" event={"ID":"0f59fa92-8480-42ba-b9e7-424c3b05b8e2","Type":"ContainerStarted","Data":"84895d7861824fc870bfeaedce7b4adde85c9646116774d1b317cb8dad0177e2"} Oct 30 21:40:40.222463 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:40.222414 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" podStartSLOduration=1.222399258 podStartE2EDuration="1.222399258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:40.217509961 +0000 UTC m=+708.813326621" watchObservedRunningTime="2026-10-30 21:40:40.222399258 +0000 UTC m=+708.818215917" Oct 30 21:40:40.398981 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:40.398891 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:40.400015 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:40.399992 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 21:40:40.400074 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:40.400044 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" podUID="0f59fa92-8480-42ba-b9e7-424c3b05b8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 21:40:40.945403 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:40.945356 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:41.326293 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:41.326265 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8eb4f19-7455v" Oct 30 21:40:41.398408 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:41.398365 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 21:40:41.398598 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:41.398425 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" podUID="0f59fa92-8480-42ba-b9e7-424c3b05b8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 21:40:41.423602 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:41.423569 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0509334-ceac-4802-8bcf-780c4cf295de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0509334-ceac-4802-8bcf-780c4cf295de-maas-api-xks-ca\") pod \"b0509334-ceac-4802-8bcf-780c4cf295de\" (UID: \"b0509334-ceac-4802-8bcf-780c4cf295de\") " Oct 30 21:40:41.423725 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:41.423612 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0509334-ceac-4802-8bcf-780c4cf295de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0509334-ceac-4802-8bcf-780c4cf295de-maas-api-openshift-ca\") pod \"b0509334-ceac-4802-8bcf-780c4cf295de\" (UID: \"b0509334-ceac-4802-8bcf-780c4cf295de\") " Oct 30 21:40:41.423725 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:41.423715 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0509334-ceac-4802-8bcf-780c4cf295de-kube-api-access-lgzmp\" (UniqueName: \"kubernetes.io/projected/b0509334-ceac-4802-8bcf-780c4cf295de-kube-api-access-lgzmp\") pod \"b0509334-ceac-4802-8bcf-780c4cf295de\" (UID: \"b0509334-ceac-4802-8bcf-780c4cf295de\") " Oct 30 21:40:41.424280 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:41.424249 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0509334-ceac-4802-8bcf-780c4cf295de-maas-api-openshift-ca" pod "b0509334-ceac-4802-8bcf-780c4cf295de" (UID: "b0509334-ceac-4802-8bcf-780c4cf295de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:41.424537 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:41.424518 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0509334-ceac-4802-8bcf-780c4cf295de-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:41.426107 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:41.426075 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0509334-ceac-4802-8bcf-780c4cf295de-maas-api-xks-ca" pod "b0509334-ceac-4802-8bcf-780c4cf295de" (UID: "b0509334-ceac-4802-8bcf-780c4cf295de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:41.426880 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:41.426855 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0509334-ceac-4802-8bcf-780c4cf295de-kube-api-access-lgzmp" pod "b0509334-ceac-4802-8bcf-780c4cf295de" (UID: "b0509334-ceac-4802-8bcf-780c4cf295de"). InnerVolumeSpecName "kube-api-access-lgzmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:41.530528 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:41.530439 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0509334-ceac-4802-8bcf-780c4cf295de-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:41.530528 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:41.530486 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgzmp\" (UniqueName: \"kubernetes.io/projected/b0509334-ceac-4802-8bcf-780c4cf295de-kube-api-access-lgzmp\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:41.750481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:41.750437 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:42.129646 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:42.129615 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8eb4f19-7455v" Oct 30 21:40:42.129646 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:42.129638 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8eb4f19-7455v" event={"ID":"b0509334-ceac-4802-8bcf-780c4cf295de","Type":"ContainerDied","Data":"5a9f6c11c6aef0014258e6770b5d8f9da2bf554b80d5f8032ea01b029b07f54b"} Oct 30 21:40:42.130109 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:42.129694 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a9f6c11c6aef0014258e6770b5d8f9da2bf554b80d5f8032ea01b029b07f54b" Oct 30 21:40:42.398950 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:42.398870 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 21:40:42.398950 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:42.398920 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" podUID="0f59fa92-8480-42ba-b9e7-424c3b05b8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 21:40:42.490989 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:42.490956 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:43.194593 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.194560 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d073a5cf-gxs6l"] Oct 30 21:40:43.195193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.195177 5801 cpu_manager.go:414] "containerMap: removing container" podUID="b0509334-ceac-4802-8bcf-780c4cf295de" containerName="revoke-keys" Oct 30 21:40:43.195246 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.195193 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0509334-ceac-4802-8bcf-780c4cf295de" containerName="revoke-keys" Oct 30 21:40:43.195310 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.195301 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0509334-ceac-4802-8bcf-780c4cf295de" containerName="revoke-keys" Oct 30 21:40:43.199002 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.198986 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d073a5cf-gxs6l" Oct 30 21:40:43.203741 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.203717 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:40:43.203741 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.203718 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:40:43.203946 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.203769 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:40:43.203946 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.203801 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:40:43.208863 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.208842 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d073a5cf-gxs6l"] Oct 30 21:40:43.239701 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.239671 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:43.386477 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.385866 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch7kx\" (UniqueName: \"kubernetes.io/projected/70ed8e91-3f4f-4fb1-9ab1-abb393956860-kube-api-access-ch7kx\") pod \"maas-api-revoke-sub-d073a5cf-gxs6l\" (UID: \"70ed8e91-3f4f-4fb1-9ab1-abb393956860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d073a5cf-gxs6l" Oct 30 21:40:43.386477 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.386073 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70ed8e91-3f4f-4fb1-9ab1-abb393956860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d073a5cf-gxs6l\" (UID: \"70ed8e91-3f4f-4fb1-9ab1-abb393956860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d073a5cf-gxs6l" Oct 30 21:40:43.386477 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.386191 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70ed8e91-3f4f-4fb1-9ab1-abb393956860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d073a5cf-gxs6l\" (UID: \"70ed8e91-3f4f-4fb1-9ab1-abb393956860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d073a5cf-gxs6l" Oct 30 21:40:43.398901 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.398865 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 21:40:43.399078 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.398927 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" podUID="0f59fa92-8480-42ba-b9e7-424c3b05b8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 21:40:43.488542 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.488456 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch7kx\" (UniqueName: \"kubernetes.io/projected/70ed8e91-3f4f-4fb1-9ab1-abb393956860-kube-api-access-ch7kx\") pod \"maas-api-revoke-sub-d073a5cf-gxs6l\" (UID: \"70ed8e91-3f4f-4fb1-9ab1-abb393956860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d073a5cf-gxs6l" Oct 30 21:40:43.488685 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.488599 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70ed8e91-3f4f-4fb1-9ab1-abb393956860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d073a5cf-gxs6l\" (UID: \"70ed8e91-3f4f-4fb1-9ab1-abb393956860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d073a5cf-gxs6l" Oct 30 21:40:43.488685 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.488673 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70ed8e91-3f4f-4fb1-9ab1-abb393956860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d073a5cf-gxs6l\" (UID: \"70ed8e91-3f4f-4fb1-9ab1-abb393956860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d073a5cf-gxs6l" Oct 30 21:40:43.489305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.489277 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70ed8e91-3f4f-4fb1-9ab1-abb393956860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d073a5cf-gxs6l\" (UID: \"70ed8e91-3f4f-4fb1-9ab1-abb393956860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d073a5cf-gxs6l" Oct 30 21:40:43.491041 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.491019 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70ed8e91-3f4f-4fb1-9ab1-abb393956860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d073a5cf-gxs6l\" (UID: \"70ed8e91-3f4f-4fb1-9ab1-abb393956860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d073a5cf-gxs6l" Oct 30 21:40:43.500946 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.500927 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch7kx\" (UniqueName: \"kubernetes.io/projected/70ed8e91-3f4f-4fb1-9ab1-abb393956860-kube-api-access-ch7kx\") pod \"maas-api-revoke-sub-d073a5cf-gxs6l\" (UID: \"70ed8e91-3f4f-4fb1-9ab1-abb393956860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d073a5cf-gxs6l" Oct 30 21:40:43.514586 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.514564 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d073a5cf-gxs6l" Oct 30 21:40:43.828135 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.828111 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d073a5cf-gxs6l"] Oct 30 21:40:43.969047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:43.969017 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:44.146894 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.146860 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d073a5cf-gxs6l" event={"ID":"70ed8e91-3f4f-4fb1-9ab1-abb393956860","Type":"ContainerStarted","Data":"e321cecfba74efba97a851000e48df5fcba695c113a6217d17dda318f1160095"} Oct 30 21:40:44.218384 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.218352 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr"] Oct 30 21:40:44.224852 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.224827 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:40:44.232477 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.232449 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1a81ae\"/\"distinct-2-w1-1a81ae-kserve-self-signed-certs\"" Oct 30 21:40:44.234968 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.234942 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr"] Oct 30 21:40:44.237029 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.237003 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:44.303828 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.303804 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-tls-certs\") pod \"distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr\" (UID: \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:40:44.303932 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.303871 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clk7h\" (UniqueName: \"kubernetes.io/projected/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-kube-api-access-clk7h\") pod \"distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr\" (UID: \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:40:44.304073 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.304047 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-model-cache\") pod \"distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr\" (UID: \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:40:44.304238 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.304142 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-tmp-dir\") pod \"distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr\" (UID: \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:40:44.304315 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.304292 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-dshm\") pod \"distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr\" (UID: \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:40:44.304387 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.304370 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-home\") pod \"distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr\" (UID: \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:40:44.403882 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.403163 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 21:40:44.403882 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.403227 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" podUID="0f59fa92-8480-42ba-b9e7-424c3b05b8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 21:40:44.408784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.407209 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-tls-certs\") pod \"distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr\" (UID: \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:40:44.408784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.407277 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clk7h\" (UniqueName: \"kubernetes.io/projected/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-kube-api-access-clk7h\") pod \"distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr\" (UID: \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:40:44.408784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.407379 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-model-cache\") pod \"distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr\" (UID: \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:40:44.408784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.407448 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-tmp-dir\") pod \"distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr\" (UID: \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:40:44.408784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.407502 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-dshm\") pod \"distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr\" (UID: \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:40:44.408784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.407547 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-home\") pod \"distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr\" (UID: \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:40:44.408784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.408033 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-home\") pod \"distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr\" (UID: \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:40:44.414782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.414696 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-model-cache\") pod \"distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr\" (UID: \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:40:44.416217 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.416194 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-tmp-dir\") pod \"distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr\" (UID: \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:40:44.416935 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.416897 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-tls-certs\") pod \"distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr\" (UID: \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:40:44.418991 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.418951 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-dshm\") pod \"distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr\" (UID: \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:40:44.428326 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.428300 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clk7h\" (UniqueName: \"kubernetes.io/projected/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-kube-api-access-clk7h\") pod \"distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr\" (UID: \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\") " pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:40:44.539615 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.539581 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:40:44.780838 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.780254 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:44.849947 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:44.849916 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr"] Oct 30 21:40:44.850609 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:40:44.850549 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2d6e8aa_9bf3_4078_99f8_e2f0be40d9b5.slice/crio-8dbee447c3662f30fcd72dec7a75186ac15f7aca6df0961d51974a6a1a072ce7 WatchSource:0}: Error finding container 8dbee447c3662f30fcd72dec7a75186ac15f7aca6df0961d51974a6a1a072ce7: Status 404 returned error can't find the container with id 8dbee447c3662f30fcd72dec7a75186ac15f7aca6df0961d51974a6a1a072ce7 Oct 30 21:40:45.180937 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:45.180897 5801 generic.go:356] "Generic (PLEG): container finished" podID="70ed8e91-3f4f-4fb1-9ab1-abb393956860" containerID="ffd196756aa7d915588f785d05c6a12581c623451450668b89add264129063fa" exitCode=0 Oct 30 21:40:45.181140 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:45.180977 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d073a5cf-gxs6l" event={"ID":"70ed8e91-3f4f-4fb1-9ab1-abb393956860","Type":"ContainerDied","Data":"ffd196756aa7d915588f785d05c6a12581c623451450668b89add264129063fa"} Oct 30 21:40:45.182743 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:45.182712 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" event={"ID":"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5","Type":"ContainerStarted","Data":"18c8ea1eabde5238d47b8c4447efbafa9ab1177f7a7cb1b04acf4832e17a12ff"} Oct 30 21:40:45.182743 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:45.182765 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" event={"ID":"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5","Type":"ContainerStarted","Data":"8dbee447c3662f30fcd72dec7a75186ac15f7aca6df0961d51974a6a1a072ce7"} Oct 30 21:40:45.223706 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:45.223661 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podStartSLOduration=1.223649829 podStartE2EDuration="1.223649829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:45.221861756 +0000 UTC m=+713.817678448" watchObservedRunningTime="2026-10-30 21:40:45.223649829 +0000 UTC m=+713.819466491" Oct 30 21:40:45.399346 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:45.399294 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 21:40:45.399795 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:45.399732 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" podUID="0f59fa92-8480-42ba-b9e7-424c3b05b8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 21:40:45.475831 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:45.474856 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:45.540708 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:45.540681 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:40:46.194281 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.194257 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3c9f153-vrtcb"] Oct 30 21:40:46.198303 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.198280 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c9f153-vrtcb" Oct 30 21:40:46.208249 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.208221 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3c9f153-vrtcb"] Oct 30 21:40:46.211842 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.211797 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:46.238789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.234925 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq78k\" (UniqueName: \"kubernetes.io/projected/c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb-kube-api-access-zq78k\") pod \"maas-api-revoke-sub-d3c9f153-vrtcb\" (UID: \"c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c9f153-vrtcb" Oct 30 21:40:46.238789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.234996 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3c9f153-vrtcb\" (UID: \"c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c9f153-vrtcb" Oct 30 21:40:46.238789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.235077 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3c9f153-vrtcb\" (UID: \"c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c9f153-vrtcb" Oct 30 21:40:46.340300 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.340257 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq78k\" (UniqueName: \"kubernetes.io/projected/c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb-kube-api-access-zq78k\") pod \"maas-api-revoke-sub-d3c9f153-vrtcb\" (UID: \"c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c9f153-vrtcb" Oct 30 21:40:46.340961 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.340933 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3c9f153-vrtcb\" (UID: \"c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c9f153-vrtcb" Oct 30 21:40:46.341080 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.341022 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3c9f153-vrtcb\" (UID: \"c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c9f153-vrtcb" Oct 30 21:40:46.341607 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.341583 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3c9f153-vrtcb\" (UID: \"c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c9f153-vrtcb" Oct 30 21:40:46.347314 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.346675 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3c9f153-vrtcb\" (UID: \"c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c9f153-vrtcb" Oct 30 21:40:46.356217 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.355942 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq78k\" (UniqueName: \"kubernetes.io/projected/c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb-kube-api-access-zq78k\") pod \"maas-api-revoke-sub-d3c9f153-vrtcb\" (UID: \"c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c9f153-vrtcb" Oct 30 21:40:46.438352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.438316 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d073a5cf-gxs6l" Oct 30 21:40:46.521129 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.521036 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c9f153-vrtcb" Oct 30 21:40:46.543503 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.543375 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70ed8e91-3f4f-4fb1-9ab1-abb393956860-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70ed8e91-3f4f-4fb1-9ab1-abb393956860-maas-api-xks-ca\") pod \"70ed8e91-3f4f-4fb1-9ab1-abb393956860\" (UID: \"70ed8e91-3f4f-4fb1-9ab1-abb393956860\") " Oct 30 21:40:46.543503 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.543476 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70ed8e91-3f4f-4fb1-9ab1-abb393956860-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70ed8e91-3f4f-4fb1-9ab1-abb393956860-maas-api-openshift-ca\") pod \"70ed8e91-3f4f-4fb1-9ab1-abb393956860\" (UID: \"70ed8e91-3f4f-4fb1-9ab1-abb393956860\") " Oct 30 21:40:46.545967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.543531 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70ed8e91-3f4f-4fb1-9ab1-abb393956860-kube-api-access-ch7kx\" (UniqueName: \"kubernetes.io/projected/70ed8e91-3f4f-4fb1-9ab1-abb393956860-kube-api-access-ch7kx\") pod \"70ed8e91-3f4f-4fb1-9ab1-abb393956860\" (UID: \"70ed8e91-3f4f-4fb1-9ab1-abb393956860\") " Oct 30 21:40:46.546234 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.546197 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70ed8e91-3f4f-4fb1-9ab1-abb393956860-maas-api-openshift-ca" pod "70ed8e91-3f4f-4fb1-9ab1-abb393956860" (UID: "70ed8e91-3f4f-4fb1-9ab1-abb393956860"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:46.546875 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.546835 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70ed8e91-3f4f-4fb1-9ab1-abb393956860-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:46.549249 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.549026 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70ed8e91-3f4f-4fb1-9ab1-abb393956860-maas-api-xks-ca" pod "70ed8e91-3f4f-4fb1-9ab1-abb393956860" (UID: "70ed8e91-3f4f-4fb1-9ab1-abb393956860"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:46.550821 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.550794 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ed8e91-3f4f-4fb1-9ab1-abb393956860-kube-api-access-ch7kx" pod "70ed8e91-3f4f-4fb1-9ab1-abb393956860" (UID: "70ed8e91-3f4f-4fb1-9ab1-abb393956860"). InnerVolumeSpecName "kube-api-access-ch7kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:46.648580 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.648549 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70ed8e91-3f4f-4fb1-9ab1-abb393956860-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:46.648580 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.648580 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch7kx\" (UniqueName: \"kubernetes.io/projected/70ed8e91-3f4f-4fb1-9ab1-abb393956860-kube-api-access-ch7kx\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:46.753004 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.752972 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:46.753554 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.753500 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:46.754370 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.754348 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:40:46.812794 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:40:46.807404 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4724fdd_3e14_4f34_8da4_ae4d7d2c20cb.slice/crio-4a5a33252c65ae2dfe32917314b0000b3ca04a347b5f84a88e915358e03b1dd0 WatchSource:0}: Error finding container 4a5a33252c65ae2dfe32917314b0000b3ca04a347b5f84a88e915358e03b1dd0: Status 404 returned error can't find the container with id 4a5a33252c65ae2dfe32917314b0000b3ca04a347b5f84a88e915358e03b1dd0 Oct 30 21:40:46.812794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:46.810324 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3c9f153-vrtcb"] Oct 30 21:40:47.185649 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:47.185613 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:47.219076 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:47.214736 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d073a5cf-gxs6l" Oct 30 21:40:47.219076 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:47.214737 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d073a5cf-gxs6l" event={"ID":"70ed8e91-3f4f-4fb1-9ab1-abb393956860","Type":"ContainerDied","Data":"e321cecfba74efba97a851000e48df5fcba695c113a6217d17dda318f1160095"} Oct 30 21:40:47.219076 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:47.215639 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e321cecfba74efba97a851000e48df5fcba695c113a6217d17dda318f1160095" Oct 30 21:40:47.222505 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:47.222289 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c9f153-vrtcb" event={"ID":"c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb","Type":"ContainerStarted","Data":"4de4fa98443d1099f1e364956f6235796c70a34af09a32cedf703ed4fc63b9da"} Oct 30 21:40:47.222505 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:47.222346 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c9f153-vrtcb" event={"ID":"c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb","Type":"ContainerStarted","Data":"4a5a33252c65ae2dfe32917314b0000b3ca04a347b5f84a88e915358e03b1dd0"} Oct 30 21:40:47.251435 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:47.251360 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c9f153-vrtcb" podStartSLOduration=1.2513371119999999 podStartE2EDuration="1.251337112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:47.246977917 +0000 UTC m=+715.842794576" watchObservedRunningTime="2026-10-30 21:40:47.251337112 +0000 UTC m=+715.847153821" Oct 30 21:40:47.555438 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:47.555405 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:40:47.568185 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:47.568158 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:40:48.034971 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:48.034889 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:48.230745 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:48.230709 5801 generic.go:356] "Generic (PLEG): container finished" podID="c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb" containerID="4de4fa98443d1099f1e364956f6235796c70a34af09a32cedf703ed4fc63b9da" exitCode=0 Oct 30 21:40:48.230961 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:48.230788 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c9f153-vrtcb" event={"ID":"c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb","Type":"ContainerDied","Data":"4de4fa98443d1099f1e364956f6235796c70a34af09a32cedf703ed4fc63b9da"} Oct 30 21:40:48.242389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:48.242363 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:40:49.039909 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:49.039875 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:49.401540 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:49.401497 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c9f153-vrtcb" Oct 30 21:40:49.495379 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:49.495345 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb-maas-api-xks-ca\") pod \"c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb\" (UID: \"c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb\") " Oct 30 21:40:49.495541 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:49.495397 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb-kube-api-access-zq78k\" (UniqueName: \"kubernetes.io/projected/c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb-kube-api-access-zq78k\") pod \"c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb\" (UID: \"c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb\") " Oct 30 21:40:49.495541 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:49.495486 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb-maas-api-openshift-ca\") pod \"c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb\" (UID: \"c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb\") " Oct 30 21:40:49.495859 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:49.495831 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb-maas-api-openshift-ca" pod "c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb" (UID: "c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:49.496086 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:49.496070 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:49.497467 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:49.497445 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb-maas-api-xks-ca" pod "c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb" (UID: "c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:49.497594 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:49.497576 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb-kube-api-access-zq78k" pod "c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb" (UID: "c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb"). InnerVolumeSpecName "kube-api-access-zq78k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:49.598637 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:49.598604 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:49.598773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:49.598643 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq78k\" (UniqueName: \"kubernetes.io/projected/c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb-kube-api-access-zq78k\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:50.101620 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:50.101577 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:50.255982 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:50.255953 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c9f153-vrtcb" Oct 30 21:40:50.255982 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:50.255962 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c9f153-vrtcb" event={"ID":"c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb","Type":"ContainerDied","Data":"4a5a33252c65ae2dfe32917314b0000b3ca04a347b5f84a88e915358e03b1dd0"} Oct 30 21:40:50.256232 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:50.256052 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a5a33252c65ae2dfe32917314b0000b3ca04a347b5f84a88e915358e03b1dd0" Oct 30 21:40:50.990778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:50.984658 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:51.990772 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:51.990271 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:52.072894 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:52.072849 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:52.197901 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:52.197855 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:52.241934 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:52.241832 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:40:52.252033 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:52.252001 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded" Oct 30 21:40:52.285868 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:52.285821 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:52.511108 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:52.511008 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:52.558776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:52.551837 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:40:52.559115 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:52.559084 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:52.616278 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:52.615833 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:52.750894 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:52.750847 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:52.806834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:52.803153 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:52.890914 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:52.890859 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:53.009323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:53.008879 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:53.087302 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:53.087206 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:53.089001 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:53.088981 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:40:53.166816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:53.166732 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:53.181685 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:53.179117 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:53.222776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:53.198271 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:53.223429 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:53.223400 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:40:53.250666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:53.250632 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:53.251423 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:53.250548 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:40:53.251868 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:53.251847 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:40:53.346818 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:53.331618 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:53.346818 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:53.341941 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:40:53.392777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:53.379372 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:40:53.392777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:53.379436 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:40:53.392777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:53.379469 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:40:53.392777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:53.386710 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:40:53.806147 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:53.806116 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:54.540333 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:54.540303 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:40:54.553131 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:54.553109 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:40:54.692328 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:54.692281 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:55.534629 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:55.534587 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:40:55.561413 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:55.561374 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:56.247847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:56.247816 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:57.044908 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:57.044874 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:57.316607 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:57.316578 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:57.841122 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:57.841083 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:58.651331 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:58.651297 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:59.119931 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.119897 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b85feb-8xhgm"] Oct 30 21:40:59.120490 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.120474 5801 cpu_manager.go:414] "containerMap: removing container" podUID="70ed8e91-3f4f-4fb1-9ab1-abb393956860" containerName="revoke-keys" Oct 30 21:40:59.120536 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.120492 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70ed8e91-3f4f-4fb1-9ab1-abb393956860" containerName="revoke-keys" Oct 30 21:40:59.120571 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.120537 5801 cpu_manager.go:414] "containerMap: removing container" podUID="c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb" containerName="revoke-keys" Oct 30 21:40:59.120571 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.120543 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb" containerName="revoke-keys" Oct 30 21:40:59.120639 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.120629 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="70ed8e91-3f4f-4fb1-9ab1-abb393956860" containerName="revoke-keys" Oct 30 21:40:59.120673 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.120646 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb" containerName="revoke-keys" Oct 30 21:40:59.125690 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.125670 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85feb-8xhgm" Oct 30 21:40:59.131103 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.131075 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:40:59.132048 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.132026 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:40:59.132166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.132097 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:40:59.132228 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.132191 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:40:59.134604 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.134582 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b85feb-8xhgm"] Oct 30 21:40:59.232320 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.232290 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88539ee2-49d3-4d50-8903-928974b64c30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3b85feb-8xhgm\" (UID: \"88539ee2-49d3-4d50-8903-928974b64c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85feb-8xhgm" Oct 30 21:40:59.232472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.232350 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88539ee2-49d3-4d50-8903-928974b64c30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3b85feb-8xhgm\" (UID: \"88539ee2-49d3-4d50-8903-928974b64c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85feb-8xhgm" Oct 30 21:40:59.232528 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.232507 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpq9q\" (UniqueName: \"kubernetes.io/projected/88539ee2-49d3-4d50-8903-928974b64c30-kube-api-access-jpq9q\") pod \"maas-api-revoke-sub-b3b85feb-8xhgm\" (UID: \"88539ee2-49d3-4d50-8903-928974b64c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85feb-8xhgm" Oct 30 21:40:59.333960 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.333925 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88539ee2-49d3-4d50-8903-928974b64c30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3b85feb-8xhgm\" (UID: \"88539ee2-49d3-4d50-8903-928974b64c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85feb-8xhgm" Oct 30 21:40:59.334137 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.333988 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88539ee2-49d3-4d50-8903-928974b64c30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3b85feb-8xhgm\" (UID: \"88539ee2-49d3-4d50-8903-928974b64c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85feb-8xhgm" Oct 30 21:40:59.334137 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.334078 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpq9q\" (UniqueName: \"kubernetes.io/projected/88539ee2-49d3-4d50-8903-928974b64c30-kube-api-access-jpq9q\") pod \"maas-api-revoke-sub-b3b85feb-8xhgm\" (UID: \"88539ee2-49d3-4d50-8903-928974b64c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85feb-8xhgm" Oct 30 21:40:59.334623 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.334603 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88539ee2-49d3-4d50-8903-928974b64c30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3b85feb-8xhgm\" (UID: \"88539ee2-49d3-4d50-8903-928974b64c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85feb-8xhgm" Oct 30 21:40:59.336259 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.336236 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88539ee2-49d3-4d50-8903-928974b64c30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3b85feb-8xhgm\" (UID: \"88539ee2-49d3-4d50-8903-928974b64c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85feb-8xhgm" Oct 30 21:40:59.346569 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.346550 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpq9q\" (UniqueName: \"kubernetes.io/projected/88539ee2-49d3-4d50-8903-928974b64c30-kube-api-access-jpq9q\") pod \"maas-api-revoke-sub-b3b85feb-8xhgm\" (UID: \"88539ee2-49d3-4d50-8903-928974b64c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85feb-8xhgm" Oct 30 21:40:59.442604 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.442534 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85feb-8xhgm" Oct 30 21:40:59.728920 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:40:59.728891 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88539ee2_49d3_4d50_8903_928974b64c30.slice/crio-05f5d89a4adad893ab8ddbf72063392ecc6c6ea7c779a37ed7dc607d17af697d WatchSource:0}: Error finding container 05f5d89a4adad893ab8ddbf72063392ecc6c6ea7c779a37ed7dc607d17af697d: Status 404 returned error can't find the container with id 05f5d89a4adad893ab8ddbf72063392ecc6c6ea7c779a37ed7dc607d17af697d Oct 30 21:40:59.729781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.729735 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b85feb-8xhgm"] Oct 30 21:40:59.743954 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:40:59.743923 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:00.278295 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:41:00.278256 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:41:00.451006 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:00.450934 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66c1025-gv9zf"] Oct 30 21:41:00.455141 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:00.455114 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66c1025-gv9zf" Oct 30 21:41:00.468456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:00.468432 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66c1025-gv9zf"] Oct 30 21:41:00.553298 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:00.553266 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff86e576-1804-4310-aee0-cc050227805d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d66c1025-gv9zf\" (UID: \"ff86e576-1804-4310-aee0-cc050227805d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66c1025-gv9zf" Oct 30 21:41:00.553514 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:00.553487 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvlzj\" (UniqueName: \"kubernetes.io/projected/ff86e576-1804-4310-aee0-cc050227805d-kube-api-access-jvlzj\") pod \"maas-api-revoke-sub-d66c1025-gv9zf\" (UID: \"ff86e576-1804-4310-aee0-cc050227805d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66c1025-gv9zf" Oct 30 21:41:00.553653 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:00.553632 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff86e576-1804-4310-aee0-cc050227805d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d66c1025-gv9zf\" (UID: \"ff86e576-1804-4310-aee0-cc050227805d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66c1025-gv9zf" Oct 30 21:41:00.567786 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:00.567745 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85feb-8xhgm" event={"ID":"88539ee2-49d3-4d50-8903-928974b64c30","Type":"ContainerStarted","Data":"f6bf890cb3b2e01c95326d09c9bf835073e258603071a396170221d35e34a3a9"} Oct 30 21:41:00.567899 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:00.567809 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85feb-8xhgm" event={"ID":"88539ee2-49d3-4d50-8903-928974b64c30","Type":"ContainerStarted","Data":"05f5d89a4adad893ab8ddbf72063392ecc6c6ea7c779a37ed7dc607d17af697d"} Oct 30 21:41:00.586337 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:00.586299 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85feb-8xhgm" podStartSLOduration=1.586288283 podStartE2EDuration="1.586288283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:00.585359022 +0000 UTC m=+729.181175685" watchObservedRunningTime="2026-10-30 21:41:00.586288283 +0000 UTC m=+729.182104943" Oct 30 21:41:00.592224 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:00.592200 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:00.655359 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:00.655113 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvlzj\" (UniqueName: \"kubernetes.io/projected/ff86e576-1804-4310-aee0-cc050227805d-kube-api-access-jvlzj\") pod \"maas-api-revoke-sub-d66c1025-gv9zf\" (UID: \"ff86e576-1804-4310-aee0-cc050227805d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66c1025-gv9zf" Oct 30 21:41:00.657229 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:00.657198 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff86e576-1804-4310-aee0-cc050227805d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d66c1025-gv9zf\" (UID: \"ff86e576-1804-4310-aee0-cc050227805d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66c1025-gv9zf" Oct 30 21:41:00.657385 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:00.657363 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff86e576-1804-4310-aee0-cc050227805d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d66c1025-gv9zf\" (UID: \"ff86e576-1804-4310-aee0-cc050227805d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66c1025-gv9zf" Oct 30 21:41:00.657858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:00.657835 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff86e576-1804-4310-aee0-cc050227805d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d66c1025-gv9zf\" (UID: \"ff86e576-1804-4310-aee0-cc050227805d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66c1025-gv9zf" Oct 30 21:41:00.662462 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:00.662435 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff86e576-1804-4310-aee0-cc050227805d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d66c1025-gv9zf\" (UID: \"ff86e576-1804-4310-aee0-cc050227805d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66c1025-gv9zf" Oct 30 21:41:00.669156 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:00.669128 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvlzj\" (UniqueName: \"kubernetes.io/projected/ff86e576-1804-4310-aee0-cc050227805d-kube-api-access-jvlzj\") pod \"maas-api-revoke-sub-d66c1025-gv9zf\" (UID: \"ff86e576-1804-4310-aee0-cc050227805d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66c1025-gv9zf" Oct 30 21:41:00.786544 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:00.786466 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66c1025-gv9zf" Oct 30 21:41:01.064707 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.064674 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d8d6034-8c89n"] Oct 30 21:41:01.069863 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.069834 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8d6034-8c89n" Oct 30 21:41:01.073901 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:41:01.073872 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff86e576_1804_4310_aee0_cc050227805d.slice/crio-57d6861aeea571c862f2eaa05eb8e08405808ba21e1089027cab9af2433efd94 WatchSource:0}: Error finding container 57d6861aeea571c862f2eaa05eb8e08405808ba21e1089027cab9af2433efd94: Status 404 returned error can't find the container with id 57d6861aeea571c862f2eaa05eb8e08405808ba21e1089027cab9af2433efd94 Oct 30 21:41:01.074433 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.074372 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66c1025-gv9zf"] Oct 30 21:41:01.092889 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.092821 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d8d6034-8c89n"] Oct 30 21:41:01.164139 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.164107 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7eb05e-10e3-42ae-ae42-9c7426701314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d8d6034-8c89n\" (UID: \"1d7eb05e-10e3-42ae-ae42-9c7426701314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8d6034-8c89n" Oct 30 21:41:01.164300 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.164271 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7eb05e-10e3-42ae-ae42-9c7426701314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d8d6034-8c89n\" (UID: \"1d7eb05e-10e3-42ae-ae42-9c7426701314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8d6034-8c89n" Oct 30 21:41:01.164375 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.164362 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r579\" (UniqueName: \"kubernetes.io/projected/1d7eb05e-10e3-42ae-ae42-9c7426701314-kube-api-access-2r579\") pod \"maas-api-revoke-sub-1d8d6034-8c89n\" (UID: \"1d7eb05e-10e3-42ae-ae42-9c7426701314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8d6034-8c89n" Oct 30 21:41:01.264624 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.264591 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81f14d1c-5765f"] Oct 30 21:41:01.265439 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.265413 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7eb05e-10e3-42ae-ae42-9c7426701314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d8d6034-8c89n\" (UID: \"1d7eb05e-10e3-42ae-ae42-9c7426701314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8d6034-8c89n" Oct 30 21:41:01.265569 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.265551 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7eb05e-10e3-42ae-ae42-9c7426701314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d8d6034-8c89n\" (UID: \"1d7eb05e-10e3-42ae-ae42-9c7426701314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8d6034-8c89n" Oct 30 21:41:01.265647 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.265633 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r579\" (UniqueName: \"kubernetes.io/projected/1d7eb05e-10e3-42ae-ae42-9c7426701314-kube-api-access-2r579\") pod \"maas-api-revoke-sub-1d8d6034-8c89n\" (UID: \"1d7eb05e-10e3-42ae-ae42-9c7426701314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8d6034-8c89n" Oct 30 21:41:01.266103 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.266076 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7eb05e-10e3-42ae-ae42-9c7426701314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d8d6034-8c89n\" (UID: \"1d7eb05e-10e3-42ae-ae42-9c7426701314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8d6034-8c89n" Oct 30 21:41:01.268034 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.268013 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7eb05e-10e3-42ae-ae42-9c7426701314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d8d6034-8c89n\" (UID: \"1d7eb05e-10e3-42ae-ae42-9c7426701314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8d6034-8c89n" Oct 30 21:41:01.270449 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.270434 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f14d1c-5765f" Oct 30 21:41:01.282332 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.282309 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81f14d1c-5765f"] Oct 30 21:41:01.283221 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.283200 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r579\" (UniqueName: \"kubernetes.io/projected/1d7eb05e-10e3-42ae-ae42-9c7426701314-kube-api-access-2r579\") pod \"maas-api-revoke-sub-1d8d6034-8c89n\" (UID: \"1d7eb05e-10e3-42ae-ae42-9c7426701314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8d6034-8c89n" Oct 30 21:41:01.366645 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.366618 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztb6h\" (UniqueName: \"kubernetes.io/projected/6ab3247a-845f-4b65-98bb-fd9752468cf6-kube-api-access-ztb6h\") pod \"maas-api-revoke-sub-81f14d1c-5765f\" (UID: \"6ab3247a-845f-4b65-98bb-fd9752468cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f14d1c-5765f" Oct 30 21:41:01.366829 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.366807 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ab3247a-845f-4b65-98bb-fd9752468cf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81f14d1c-5765f\" (UID: \"6ab3247a-845f-4b65-98bb-fd9752468cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f14d1c-5765f" Oct 30 21:41:01.366906 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.366867 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ab3247a-845f-4b65-98bb-fd9752468cf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81f14d1c-5765f\" (UID: \"6ab3247a-845f-4b65-98bb-fd9752468cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f14d1c-5765f" Oct 30 21:41:01.396626 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.396599 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8d6034-8c89n" Oct 30 21:41:01.468933 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.468902 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztb6h\" (UniqueName: \"kubernetes.io/projected/6ab3247a-845f-4b65-98bb-fd9752468cf6-kube-api-access-ztb6h\") pod \"maas-api-revoke-sub-81f14d1c-5765f\" (UID: \"6ab3247a-845f-4b65-98bb-fd9752468cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f14d1c-5765f" Oct 30 21:41:01.469101 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.468993 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ab3247a-845f-4b65-98bb-fd9752468cf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81f14d1c-5765f\" (UID: \"6ab3247a-845f-4b65-98bb-fd9752468cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f14d1c-5765f" Oct 30 21:41:01.469101 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.469052 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ab3247a-845f-4b65-98bb-fd9752468cf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81f14d1c-5765f\" (UID: \"6ab3247a-845f-4b65-98bb-fd9752468cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f14d1c-5765f" Oct 30 21:41:01.470050 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.470024 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ab3247a-845f-4b65-98bb-fd9752468cf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81f14d1c-5765f\" (UID: \"6ab3247a-845f-4b65-98bb-fd9752468cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f14d1c-5765f" Oct 30 21:41:01.471960 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.471939 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ab3247a-845f-4b65-98bb-fd9752468cf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81f14d1c-5765f\" (UID: \"6ab3247a-845f-4b65-98bb-fd9752468cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f14d1c-5765f" Oct 30 21:41:01.482537 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.482501 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztb6h\" (UniqueName: \"kubernetes.io/projected/6ab3247a-845f-4b65-98bb-fd9752468cf6-kube-api-access-ztb6h\") pod \"maas-api-revoke-sub-81f14d1c-5765f\" (UID: \"6ab3247a-845f-4b65-98bb-fd9752468cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f14d1c-5765f" Oct 30 21:41:01.547123 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.547091 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:01.575812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.575782 5801 generic.go:356] "Generic (PLEG): container finished" podID="88539ee2-49d3-4d50-8903-928974b64c30" containerID="f6bf890cb3b2e01c95326d09c9bf835073e258603071a396170221d35e34a3a9" exitCode=0 Oct 30 21:41:01.575936 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.575894 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85feb-8xhgm" event={"ID":"88539ee2-49d3-4d50-8903-928974b64c30","Type":"ContainerDied","Data":"f6bf890cb3b2e01c95326d09c9bf835073e258603071a396170221d35e34a3a9"} Oct 30 21:41:01.577394 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.577372 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66c1025-gv9zf" event={"ID":"ff86e576-1804-4310-aee0-cc050227805d","Type":"ContainerStarted","Data":"4e89804aa8bb4bb619f493a3e195853e0d6357a6c00154f79e110d44fed56a3e"} Oct 30 21:41:01.577506 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.577421 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66c1025-gv9zf" event={"ID":"ff86e576-1804-4310-aee0-cc050227805d","Type":"ContainerStarted","Data":"57d6861aeea571c862f2eaa05eb8e08405808ba21e1089027cab9af2433efd94"} Oct 30 21:41:01.585976 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.585953 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f14d1c-5765f" Oct 30 21:41:01.618876 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.618784 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66c1025-gv9zf" podStartSLOduration=1.618771355 podStartE2EDuration="1.618771355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:01.616116993 +0000 UTC m=+730.211933667" watchObservedRunningTime="2026-10-30 21:41:01.618771355 +0000 UTC m=+730.214588011" Oct 30 21:41:01.692240 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.692208 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d8d6034-8c89n"] Oct 30 21:41:01.895945 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:41:01.895913 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab3247a_845f_4b65_98bb_fd9752468cf6.slice/crio-be480c18c407b20cac4957eead1d8c9396bd5d2c2090e4cc47f22c7f3afc754b WatchSource:0}: Error finding container be480c18c407b20cac4957eead1d8c9396bd5d2c2090e4cc47f22c7f3afc754b: Status 404 returned error can't find the container with id be480c18c407b20cac4957eead1d8c9396bd5d2c2090e4cc47f22c7f3afc754b Oct 30 21:41:01.897473 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:01.897450 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81f14d1c-5765f"] Oct 30 21:41:02.325596 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:02.325557 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:02.586740 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:02.586704 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8d6034-8c89n" event={"ID":"1d7eb05e-10e3-42ae-ae42-9c7426701314","Type":"ContainerStarted","Data":"ed6ab96c86b45f59aaefd168e53ceca5b3fbdca3dc57bf4e4f1035c281c8b1d9"} Oct 30 21:41:02.586944 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:02.586796 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8d6034-8c89n" event={"ID":"1d7eb05e-10e3-42ae-ae42-9c7426701314","Type":"ContainerStarted","Data":"8a4b6853270cdcc4dc5d4821ff85cdb27b8ffdf776e1e972f60e4a095fa9015c"} Oct 30 21:41:02.588549 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:02.588524 5801 generic.go:356] "Generic (PLEG): container finished" podID="ff86e576-1804-4310-aee0-cc050227805d" containerID="4e89804aa8bb4bb619f493a3e195853e0d6357a6c00154f79e110d44fed56a3e" exitCode=0 Oct 30 21:41:02.588675 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:02.588586 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66c1025-gv9zf" event={"ID":"ff86e576-1804-4310-aee0-cc050227805d","Type":"ContainerDied","Data":"4e89804aa8bb4bb619f493a3e195853e0d6357a6c00154f79e110d44fed56a3e"} Oct 30 21:41:02.590566 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:02.590409 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f14d1c-5765f" event={"ID":"6ab3247a-845f-4b65-98bb-fd9752468cf6","Type":"ContainerStarted","Data":"6586bf5e49242be85cae4a8cd11d786ff7f021e3114d055347a38a1657e778bd"} Oct 30 21:41:02.590566 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:02.590457 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f14d1c-5765f" event={"ID":"6ab3247a-845f-4b65-98bb-fd9752468cf6","Type":"ContainerStarted","Data":"be480c18c407b20cac4957eead1d8c9396bd5d2c2090e4cc47f22c7f3afc754b"} Oct 30 21:41:02.605550 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:02.605502 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8d6034-8c89n" podStartSLOduration=1.605490537 podStartE2EDuration="1.605490537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:02.604461256 +0000 UTC m=+731.200277937" watchObservedRunningTime="2026-10-30 21:41:02.605490537 +0000 UTC m=+731.201307197" Oct 30 21:41:02.645245 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:02.645193 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f14d1c-5765f" podStartSLOduration=1.645176133 podStartE2EDuration="1.645176133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:02.640059118 +0000 UTC m=+731.235875779" watchObservedRunningTime="2026-10-30 21:41:02.645176133 +0000 UTC m=+731.240992792" Oct 30 21:41:02.739983 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:02.739960 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85feb-8xhgm" Oct 30 21:41:02.786534 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:02.786497 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88539ee2-49d3-4d50-8903-928974b64c30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88539ee2-49d3-4d50-8903-928974b64c30-maas-api-openshift-ca\") pod \"88539ee2-49d3-4d50-8903-928974b64c30\" (UID: \"88539ee2-49d3-4d50-8903-928974b64c30\") " Oct 30 21:41:02.786687 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:02.786656 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88539ee2-49d3-4d50-8903-928974b64c30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88539ee2-49d3-4d50-8903-928974b64c30-maas-api-xks-ca\") pod \"88539ee2-49d3-4d50-8903-928974b64c30\" (UID: \"88539ee2-49d3-4d50-8903-928974b64c30\") " Oct 30 21:41:02.786767 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:02.786725 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88539ee2-49d3-4d50-8903-928974b64c30-kube-api-access-jpq9q\" (UniqueName: \"kubernetes.io/projected/88539ee2-49d3-4d50-8903-928974b64c30-kube-api-access-jpq9q\") pod \"88539ee2-49d3-4d50-8903-928974b64c30\" (UID: \"88539ee2-49d3-4d50-8903-928974b64c30\") " Oct 30 21:41:02.786935 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:02.786902 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88539ee2-49d3-4d50-8903-928974b64c30-maas-api-openshift-ca" pod "88539ee2-49d3-4d50-8903-928974b64c30" (UID: "88539ee2-49d3-4d50-8903-928974b64c30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:02.787240 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:02.787216 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88539ee2-49d3-4d50-8903-928974b64c30-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:02.788834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:02.788805 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88539ee2-49d3-4d50-8903-928974b64c30-maas-api-xks-ca" pod "88539ee2-49d3-4d50-8903-928974b64c30" (UID: "88539ee2-49d3-4d50-8903-928974b64c30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:02.788939 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:02.788880 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88539ee2-49d3-4d50-8903-928974b64c30-kube-api-access-jpq9q" pod "88539ee2-49d3-4d50-8903-928974b64c30" (UID: "88539ee2-49d3-4d50-8903-928974b64c30"). InnerVolumeSpecName "kube-api-access-jpq9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:02.888803 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:02.888775 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpq9q\" (UniqueName: \"kubernetes.io/projected/88539ee2-49d3-4d50-8903-928974b64c30-kube-api-access-jpq9q\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:02.888803 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:02.888804 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88539ee2-49d3-4d50-8903-928974b64c30-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:03.046249 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:03.046208 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:03.599033 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:03.598958 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85feb-8xhgm" event={"ID":"88539ee2-49d3-4d50-8903-928974b64c30","Type":"ContainerDied","Data":"05f5d89a4adad893ab8ddbf72063392ecc6c6ea7c779a37ed7dc607d17af697d"} Oct 30 21:41:03.599033 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:03.599005 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05f5d89a4adad893ab8ddbf72063392ecc6c6ea7c779a37ed7dc607d17af697d" Oct 30 21:41:03.599033 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:03.598969 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b85feb-8xhgm" Oct 30 21:41:03.600699 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:03.600669 5801 generic.go:356] "Generic (PLEG): container finished" podID="1d7eb05e-10e3-42ae-ae42-9c7426701314" containerID="ed6ab96c86b45f59aaefd168e53ceca5b3fbdca3dc57bf4e4f1035c281c8b1d9" exitCode=0 Oct 30 21:41:03.600854 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:03.600779 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8d6034-8c89n" event={"ID":"1d7eb05e-10e3-42ae-ae42-9c7426701314","Type":"ContainerDied","Data":"ed6ab96c86b45f59aaefd168e53ceca5b3fbdca3dc57bf4e4f1035c281c8b1d9"} Oct 30 21:41:03.602589 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:03.602569 5801 generic.go:356] "Generic (PLEG): container finished" podID="6ab3247a-845f-4b65-98bb-fd9752468cf6" containerID="6586bf5e49242be85cae4a8cd11d786ff7f021e3114d055347a38a1657e778bd" exitCode=0 Oct 30 21:41:03.602672 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:03.602628 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f14d1c-5765f" event={"ID":"6ab3247a-845f-4b65-98bb-fd9752468cf6","Type":"ContainerDied","Data":"6586bf5e49242be85cae4a8cd11d786ff7f021e3114d055347a38a1657e778bd"} Oct 30 21:41:03.763120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:03.763084 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66c1025-gv9zf" Oct 30 21:41:03.783844 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:03.783814 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:03.901601 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:03.901495 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff86e576-1804-4310-aee0-cc050227805d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff86e576-1804-4310-aee0-cc050227805d-maas-api-xks-ca\") pod \"ff86e576-1804-4310-aee0-cc050227805d\" (UID: \"ff86e576-1804-4310-aee0-cc050227805d\") " Oct 30 21:41:03.901601 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:03.901572 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff86e576-1804-4310-aee0-cc050227805d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff86e576-1804-4310-aee0-cc050227805d-maas-api-openshift-ca\") pod \"ff86e576-1804-4310-aee0-cc050227805d\" (UID: \"ff86e576-1804-4310-aee0-cc050227805d\") " Oct 30 21:41:03.901877 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:03.901812 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff86e576-1804-4310-aee0-cc050227805d-kube-api-access-jvlzj\" (UniqueName: \"kubernetes.io/projected/ff86e576-1804-4310-aee0-cc050227805d-kube-api-access-jvlzj\") pod \"ff86e576-1804-4310-aee0-cc050227805d\" (UID: \"ff86e576-1804-4310-aee0-cc050227805d\") " Oct 30 21:41:03.902009 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:03.901985 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff86e576-1804-4310-aee0-cc050227805d-maas-api-openshift-ca" pod "ff86e576-1804-4310-aee0-cc050227805d" (UID: "ff86e576-1804-4310-aee0-cc050227805d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:03.902416 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:03.902394 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff86e576-1804-4310-aee0-cc050227805d-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:03.903739 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:03.903715 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff86e576-1804-4310-aee0-cc050227805d-maas-api-xks-ca" pod "ff86e576-1804-4310-aee0-cc050227805d" (UID: "ff86e576-1804-4310-aee0-cc050227805d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:03.903972 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:03.903957 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff86e576-1804-4310-aee0-cc050227805d-kube-api-access-jvlzj" pod "ff86e576-1804-4310-aee0-cc050227805d" (UID: "ff86e576-1804-4310-aee0-cc050227805d"). InnerVolumeSpecName "kube-api-access-jvlzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:04.003237 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:04.003199 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff86e576-1804-4310-aee0-cc050227805d-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:04.003237 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:04.003231 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvlzj\" (UniqueName: \"kubernetes.io/projected/ff86e576-1804-4310-aee0-cc050227805d-kube-api-access-jvlzj\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:04.546627 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:04.546595 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:04.614610 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:04.614583 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66c1025-gv9zf" Oct 30 21:41:04.614610 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:04.614578 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66c1025-gv9zf" event={"ID":"ff86e576-1804-4310-aee0-cc050227805d","Type":"ContainerDied","Data":"57d6861aeea571c862f2eaa05eb8e08405808ba21e1089027cab9af2433efd94"} Oct 30 21:41:04.614851 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:04.614638 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57d6861aeea571c862f2eaa05eb8e08405808ba21e1089027cab9af2433efd94" Oct 30 21:41:04.809116 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:04.809090 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8d6034-8c89n" Oct 30 21:41:04.842286 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:04.842256 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f14d1c-5765f" Oct 30 21:41:04.917464 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:04.917434 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d7eb05e-10e3-42ae-ae42-9c7426701314-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7eb05e-10e3-42ae-ae42-9c7426701314-maas-api-xks-ca\") pod \"1d7eb05e-10e3-42ae-ae42-9c7426701314\" (UID: \"1d7eb05e-10e3-42ae-ae42-9c7426701314\") " Oct 30 21:41:04.917656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:04.917509 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ab3247a-845f-4b65-98bb-fd9752468cf6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ab3247a-845f-4b65-98bb-fd9752468cf6-maas-api-xks-ca\") pod \"6ab3247a-845f-4b65-98bb-fd9752468cf6\" (UID: \"6ab3247a-845f-4b65-98bb-fd9752468cf6\") " Oct 30 21:41:04.917727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:04.917673 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d7eb05e-10e3-42ae-ae42-9c7426701314-kube-api-access-2r579\" (UniqueName: \"kubernetes.io/projected/1d7eb05e-10e3-42ae-ae42-9c7426701314-kube-api-access-2r579\") pod \"1d7eb05e-10e3-42ae-ae42-9c7426701314\" (UID: \"1d7eb05e-10e3-42ae-ae42-9c7426701314\") " Oct 30 21:41:04.917727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:04.917717 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d7eb05e-10e3-42ae-ae42-9c7426701314-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7eb05e-10e3-42ae-ae42-9c7426701314-maas-api-openshift-ca\") pod \"1d7eb05e-10e3-42ae-ae42-9c7426701314\" (UID: \"1d7eb05e-10e3-42ae-ae42-9c7426701314\") " Oct 30 21:41:04.917875 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:04.917749 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ab3247a-845f-4b65-98bb-fd9752468cf6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ab3247a-845f-4b65-98bb-fd9752468cf6-maas-api-openshift-ca\") pod \"6ab3247a-845f-4b65-98bb-fd9752468cf6\" (UID: \"6ab3247a-845f-4b65-98bb-fd9752468cf6\") " Oct 30 21:41:04.917995 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:04.917980 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab3247a-845f-4b65-98bb-fd9752468cf6-kube-api-access-ztb6h\" (UniqueName: \"kubernetes.io/projected/6ab3247a-845f-4b65-98bb-fd9752468cf6-kube-api-access-ztb6h\") pod \"6ab3247a-845f-4b65-98bb-fd9752468cf6\" (UID: \"6ab3247a-845f-4b65-98bb-fd9752468cf6\") " Oct 30 21:41:04.918143 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:04.918114 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d7eb05e-10e3-42ae-ae42-9c7426701314-maas-api-openshift-ca" pod "1d7eb05e-10e3-42ae-ae42-9c7426701314" (UID: "1d7eb05e-10e3-42ae-ae42-9c7426701314"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:04.918346 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:04.918188 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab3247a-845f-4b65-98bb-fd9752468cf6-maas-api-openshift-ca" pod "6ab3247a-845f-4b65-98bb-fd9752468cf6" (UID: "6ab3247a-845f-4b65-98bb-fd9752468cf6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:04.918661 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:04.918636 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7eb05e-10e3-42ae-ae42-9c7426701314-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:04.918740 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:04.918666 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ab3247a-845f-4b65-98bb-fd9752468cf6-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:04.919816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:04.919789 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ab3247a-845f-4b65-98bb-fd9752468cf6-maas-api-xks-ca" pod "6ab3247a-845f-4b65-98bb-fd9752468cf6" (UID: "6ab3247a-845f-4b65-98bb-fd9752468cf6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:04.919933 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:04.919886 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d7eb05e-10e3-42ae-ae42-9c7426701314-maas-api-xks-ca" pod "1d7eb05e-10e3-42ae-ae42-9c7426701314" (UID: "1d7eb05e-10e3-42ae-ae42-9c7426701314"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:04.920045 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:04.920023 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7eb05e-10e3-42ae-ae42-9c7426701314-kube-api-access-2r579" pod "1d7eb05e-10e3-42ae-ae42-9c7426701314" (UID: "1d7eb05e-10e3-42ae-ae42-9c7426701314"). InnerVolumeSpecName "kube-api-access-2r579". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:04.920109 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:04.920039 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab3247a-845f-4b65-98bb-fd9752468cf6-kube-api-access-ztb6h" pod "6ab3247a-845f-4b65-98bb-fd9752468cf6" (UID: "6ab3247a-845f-4b65-98bb-fd9752468cf6"). InnerVolumeSpecName "kube-api-access-ztb6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:05.020135 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:05.020104 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztb6h\" (UniqueName: \"kubernetes.io/projected/6ab3247a-845f-4b65-98bb-fd9752468cf6-kube-api-access-ztb6h\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:05.020135 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:05.020132 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7eb05e-10e3-42ae-ae42-9c7426701314-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:05.020135 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:05.020142 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ab3247a-845f-4b65-98bb-fd9752468cf6-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:05.020333 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:05.020151 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r579\" (UniqueName: \"kubernetes.io/projected/1d7eb05e-10e3-42ae-ae42-9c7426701314-kube-api-access-2r579\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:05.335294 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:05.335255 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:05.623141 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:05.623071 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8d6034-8c89n" Oct 30 21:41:05.623141 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:05.623100 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8d6034-8c89n" event={"ID":"1d7eb05e-10e3-42ae-ae42-9c7426701314","Type":"ContainerDied","Data":"8a4b6853270cdcc4dc5d4821ff85cdb27b8ffdf776e1e972f60e4a095fa9015c"} Oct 30 21:41:05.623643 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:05.623145 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a4b6853270cdcc4dc5d4821ff85cdb27b8ffdf776e1e972f60e4a095fa9015c" Oct 30 21:41:05.624581 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:05.624561 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f14d1c-5765f" Oct 30 21:41:05.624696 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:05.624591 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f14d1c-5765f" event={"ID":"6ab3247a-845f-4b65-98bb-fd9752468cf6","Type":"ContainerDied","Data":"be480c18c407b20cac4957eead1d8c9396bd5d2c2090e4cc47f22c7f3afc754b"} Oct 30 21:41:05.624696 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:05.624631 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be480c18c407b20cac4957eead1d8c9396bd5d2c2090e4cc47f22c7f3afc754b" Oct 30 21:41:06.036558 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:06.036459 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:06.744187 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:06.744151 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:06.797547 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:06.797507 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2eda136-jh2tv"] Oct 30 21:41:06.798791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:06.798747 5801 cpu_manager.go:414] "containerMap: removing container" podUID="ff86e576-1804-4310-aee0-cc050227805d" containerName="revoke-keys" Oct 30 21:41:06.798791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:06.798789 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff86e576-1804-4310-aee0-cc050227805d" containerName="revoke-keys" Oct 30 21:41:06.799011 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:06.798802 5801 cpu_manager.go:414] "containerMap: removing container" podUID="1d7eb05e-10e3-42ae-ae42-9c7426701314" containerName="revoke-keys" Oct 30 21:41:06.799011 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:06.798812 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d7eb05e-10e3-42ae-ae42-9c7426701314" containerName="revoke-keys" Oct 30 21:41:06.799011 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:06.798824 5801 cpu_manager.go:414] "containerMap: removing container" podUID="6ab3247a-845f-4b65-98bb-fd9752468cf6" containerName="revoke-keys" Oct 30 21:41:06.799011 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:06.798831 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab3247a-845f-4b65-98bb-fd9752468cf6" containerName="revoke-keys" Oct 30 21:41:06.799011 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:06.798844 5801 cpu_manager.go:414] "containerMap: removing container" podUID="88539ee2-49d3-4d50-8903-928974b64c30" containerName="revoke-keys" Oct 30 21:41:06.799011 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:06.798851 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88539ee2-49d3-4d50-8903-928974b64c30" containerName="revoke-keys" Oct 30 21:41:06.799011 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:06.798986 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff86e576-1804-4310-aee0-cc050227805d" containerName="revoke-keys" Oct 30 21:41:06.799011 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:06.798995 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d7eb05e-10e3-42ae-ae42-9c7426701314" containerName="revoke-keys" Oct 30 21:41:06.799011 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:06.799007 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab3247a-845f-4b65-98bb-fd9752468cf6" containerName="revoke-keys" Oct 30 21:41:06.799011 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:06.799016 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="88539ee2-49d3-4d50-8903-928974b64c30" containerName="revoke-keys" Oct 30 21:41:06.803683 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:06.803661 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2eda136-jh2tv" Oct 30 21:41:06.811110 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:06.811087 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2eda136-jh2tv"] Oct 30 21:41:06.811386 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:06.811367 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:41:06.811472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:06.811384 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:41:06.811472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:06.811391 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:41:06.811581 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:06.811536 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:41:06.946221 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:06.946189 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7836e324-455e-47b4-96df-586ef9cfd53e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2eda136-jh2tv\" (UID: \"7836e324-455e-47b4-96df-586ef9cfd53e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2eda136-jh2tv" Oct 30 21:41:06.946396 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:06.946261 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjlqx\" (UniqueName: \"kubernetes.io/projected/7836e324-455e-47b4-96df-586ef9cfd53e-kube-api-access-vjlqx\") pod \"maas-api-revoke-sub-b2eda136-jh2tv\" (UID: \"7836e324-455e-47b4-96df-586ef9cfd53e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2eda136-jh2tv" Oct 30 21:41:06.946443 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:06.946426 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7836e324-455e-47b4-96df-586ef9cfd53e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2eda136-jh2tv\" (UID: \"7836e324-455e-47b4-96df-586ef9cfd53e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2eda136-jh2tv" Oct 30 21:41:07.047685 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:07.047607 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7836e324-455e-47b4-96df-586ef9cfd53e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2eda136-jh2tv\" (UID: \"7836e324-455e-47b4-96df-586ef9cfd53e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2eda136-jh2tv" Oct 30 21:41:07.047840 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:07.047732 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7836e324-455e-47b4-96df-586ef9cfd53e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2eda136-jh2tv\" (UID: \"7836e324-455e-47b4-96df-586ef9cfd53e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2eda136-jh2tv" Oct 30 21:41:07.047840 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:07.047826 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjlqx\" (UniqueName: \"kubernetes.io/projected/7836e324-455e-47b4-96df-586ef9cfd53e-kube-api-access-vjlqx\") pod \"maas-api-revoke-sub-b2eda136-jh2tv\" (UID: \"7836e324-455e-47b4-96df-586ef9cfd53e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2eda136-jh2tv" Oct 30 21:41:07.048251 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:07.048230 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7836e324-455e-47b4-96df-586ef9cfd53e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2eda136-jh2tv\" (UID: \"7836e324-455e-47b4-96df-586ef9cfd53e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2eda136-jh2tv" Oct 30 21:41:07.050090 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:07.050062 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7836e324-455e-47b4-96df-586ef9cfd53e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2eda136-jh2tv\" (UID: \"7836e324-455e-47b4-96df-586ef9cfd53e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2eda136-jh2tv" Oct 30 21:41:07.062581 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:07.062555 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjlqx\" (UniqueName: \"kubernetes.io/projected/7836e324-455e-47b4-96df-586ef9cfd53e-kube-api-access-vjlqx\") pod \"maas-api-revoke-sub-b2eda136-jh2tv\" (UID: \"7836e324-455e-47b4-96df-586ef9cfd53e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2eda136-jh2tv" Oct 30 21:41:07.147212 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:07.147181 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2eda136-jh2tv" Oct 30 21:41:07.430301 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:41:07.430264 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7836e324_455e_47b4_96df_586ef9cfd53e.slice/crio-f415ac6ecc2c5f1f53cc257d929028869b886cc767f9f13982378ba7970acd4b WatchSource:0}: Error finding container f415ac6ecc2c5f1f53cc257d929028869b886cc767f9f13982378ba7970acd4b: Status 404 returned error can't find the container with id f415ac6ecc2c5f1f53cc257d929028869b886cc767f9f13982378ba7970acd4b Oct 30 21:41:07.430794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:07.430771 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2eda136-jh2tv"] Oct 30 21:41:07.447110 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:07.447067 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:07.640948 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:07.640906 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2eda136-jh2tv" event={"ID":"7836e324-455e-47b4-96df-586ef9cfd53e","Type":"ContainerStarted","Data":"f415ac6ecc2c5f1f53cc257d929028869b886cc767f9f13982378ba7970acd4b"} Oct 30 21:41:08.130799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.130737 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:08.333843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.333808 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2"] Oct 30 21:41:08.338839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.338808 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.368007 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.367970 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2"] Oct 30 21:41:08.451129 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.451046 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl"] Oct 30 21:41:08.457063 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.457012 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.472101 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.471793 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl"] Oct 30 21:41:08.473207 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.473178 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-credential-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.473365 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.473249 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-data\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.473506 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.473478 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3895a8-b605-453e-8b79-93b54f506dfb-istiod-ca-cert\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.473561 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.473532 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-workload-certs\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.473602 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.473558 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-envoy\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.473602 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.473592 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-token\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.473678 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.473657 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khd78\" (UniqueName: \"kubernetes.io/projected/9e3895a8-b605-453e-8b79-93b54f506dfb-kube-api-access-khd78\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.473821 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.473803 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-workload-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.473879 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.473863 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-podinfo\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.480229 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.480205 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl"] Oct 30 21:41:08.490473 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.490452 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.493894 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.492493 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl"] Oct 30 21:41:08.575443 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.575404 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-podinfo\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.575611 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.575471 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-workload-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.575611 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.575565 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3895a8-b605-453e-8b79-93b54f506dfb-istiod-ca-cert\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.575611 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.575597 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-envoy\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.575802 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.575625 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-data\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.575802 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.575642 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istiod-ca-cert\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.575802 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.575795 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-envoy\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.575971 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.575816 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-podinfo\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.575971 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.575842 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-podinfo\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.575971 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.575866 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-data\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.575971 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.575942 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-credential-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.576177 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.576003 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcm4b\" (UniqueName: \"kubernetes.io/projected/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-kube-api-access-mcm4b\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.576177 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.576031 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-credential-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.576177 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.576109 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw775\" (UniqueName: \"kubernetes.io/projected/c1ee1395-263a-4afa-8f55-244a8edcd3a8-kube-api-access-dw775\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.576177 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.576143 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-data\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.576177 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.576142 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-data\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.576408 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.576208 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-token\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.576408 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.576255 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-workload-certs\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.576408 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.576289 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-credential-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.576408 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.576288 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3895a8-b605-453e-8b79-93b54f506dfb-istiod-ca-cert\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.576408 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.576293 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-token\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.576408 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.576357 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-token\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.576408 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.576405 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khd78\" (UniqueName: \"kubernetes.io/projected/9e3895a8-b605-453e-8b79-93b54f506dfb-kube-api-access-khd78\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.576781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.576495 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-envoy\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.576781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.576544 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-workload-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.576781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.576568 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-workload-certs\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.576781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.576600 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-workload-certs\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.576781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.576643 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-credential-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.576781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.576661 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-workload-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.576781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.576722 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-workload-certs\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.576781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.576785 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istiod-ca-cert\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.577190 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.576818 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-workload-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.578162 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.578142 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-envoy\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.578397 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.578376 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-podinfo\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.588944 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.588912 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khd78\" (UniqueName: \"kubernetes.io/projected/9e3895a8-b605-453e-8b79-93b54f506dfb-kube-api-access-khd78\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.592335 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.592306 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-token\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.650010 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.649976 5801 generic.go:356] "Generic (PLEG): container finished" podID="7836e324-455e-47b4-96df-586ef9cfd53e" containerID="6c21f90d21a6d01e3f9686c6d1be6e75d21d4fa0b43e94a2ffcfd65ea0b5129a" exitCode=0 Oct 30 21:41:08.650168 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.650011 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2eda136-jh2tv" event={"ID":"7836e324-455e-47b4-96df-586ef9cfd53e","Type":"ContainerDied","Data":"6c21f90d21a6d01e3f9686c6d1be6e75d21d4fa0b43e94a2ffcfd65ea0b5129a"} Oct 30 21:41:08.672611 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.672567 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:08.678370 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.678349 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-envoy\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.678458 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.678399 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-workload-certs\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.678458 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.678430 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-credential-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.678458 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.678451 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-workload-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.678613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.678478 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-workload-certs\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.678613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.678511 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istiod-ca-cert\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.678613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.678537 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-podinfo\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.678613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.678591 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-workload-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.678790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.678682 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-data\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.678790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.678710 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istiod-ca-cert\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.678907 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.678882 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-workload-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.678907 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.678896 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-credential-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.679018 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.678922 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-envoy\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.679018 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.678990 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-workload-certs\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.679617 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.679510 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istiod-ca-cert\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.679617 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.679510 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-workload-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.679617 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.679558 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istiod-ca-cert\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.679617 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.679567 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-podinfo\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.679983 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.679787 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-workload-certs\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.679983 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.679793 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-data\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.679983 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.679940 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcm4b\" (UniqueName: \"kubernetes.io/projected/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-kube-api-access-mcm4b\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.679983 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.679971 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-credential-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.680186 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.680015 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw775\" (UniqueName: \"kubernetes.io/projected/c1ee1395-263a-4afa-8f55-244a8edcd3a8-kube-api-access-dw775\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.680186 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.680051 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-data\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.680186 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.680103 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-token\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.680186 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.680128 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-token\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.680777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.680490 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-data\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.680874 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.680784 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-credential-socket\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.681239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.681219 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-envoy\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.681526 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.681507 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-envoy\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.681787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.681738 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-podinfo\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.682343 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.682324 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-podinfo\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.695175 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.695147 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-token\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.695319 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.695233 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcm4b\" (UniqueName: \"kubernetes.io/projected/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-kube-api-access-mcm4b\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.695392 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.695320 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw775\" (UniqueName: \"kubernetes.io/projected/c1ee1395-263a-4afa-8f55-244a8edcd3a8-kube-api-access-dw775\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.695573 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.695552 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-token\") pod \"e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.796574 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.796528 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k"] Oct 30 21:41:08.802698 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.802668 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:08.819451 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.819416 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k"] Oct 30 21:41:08.834872 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.834844 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:08.840340 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.840316 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:08.968270 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.968237 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2"] Oct 30 21:41:08.972391 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:41:08.972358 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3895a8_b605_453e_8b79_93b54f506dfb.slice/crio-ae3cf7a868203a39f8bb0ece30492cfe0835e8672bc3db16fbbdc4effad108c5 WatchSource:0}: Error finding container ae3cf7a868203a39f8bb0ece30492cfe0835e8672bc3db16fbbdc4effad108c5: Status 404 returned error can't find the container with id ae3cf7a868203a39f8bb0ece30492cfe0835e8672bc3db16fbbdc4effad108c5 Oct 30 21:41:08.975483 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.975444 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:08.975581 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.975536 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:08.975620 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.975584 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:08.987956 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.987930 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-token\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:08.988062 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.988016 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-envoy\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:08.988062 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.988055 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-workload-certs\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:08.988179 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.988160 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istiod-ca-cert\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:08.988235 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.988201 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-data\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:08.988285 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.988265 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cddz\" (UniqueName: \"kubernetes.io/projected/b99efd6a-6f7d-444a-89dd-bf292349a7cc-kube-api-access-5cddz\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:08.988337 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.988307 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-workload-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:08.988398 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.988348 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-podinfo\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:08.988521 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:08.988493 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-credential-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:09.045631 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.045598 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:09.089505 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.089474 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-data\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:09.089691 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.089522 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cddz\" (UniqueName: \"kubernetes.io/projected/b99efd6a-6f7d-444a-89dd-bf292349a7cc-kube-api-access-5cddz\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:09.089691 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.089580 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-workload-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:09.089691 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.089645 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-podinfo\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:09.089912 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.089705 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-credential-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:09.089912 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.089817 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-token\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:09.089912 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.089850 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-envoy\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:09.089912 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.089891 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-workload-certs\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:09.090110 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.089953 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-data\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:09.090110 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.090028 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istiod-ca-cert\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:09.093085 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.090737 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istiod-ca-cert\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:09.093085 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.091303 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-workload-certs\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:09.093085 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.091625 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-credential-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:09.093085 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.091902 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-workload-socket\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:09.094147 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.094124 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-podinfo\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:09.094265 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.094164 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-envoy\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:09.106726 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.106646 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-token\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:09.107422 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.107351 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cddz\" (UniqueName: \"kubernetes.io/projected/b99efd6a-6f7d-444a-89dd-bf292349a7cc-kube-api-access-5cddz\") pod \"e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:09.131626 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.131590 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:09.142922 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.142883 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl"] Oct 30 21:41:09.175118 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.175030 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl"] Oct 30 21:41:09.176319 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.176154 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:09.176702 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.176585 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:09.176980 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.176954 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:09.204623 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.204564 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:09.204804 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.204653 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:09.204804 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.204688 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:09.479437 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.479407 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k"] Oct 30 21:41:09.483365 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.483329 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:09.483477 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.483410 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:09.483546 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.483486 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:09.704292 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.704254 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" event={"ID":"b99efd6a-6f7d-444a-89dd-bf292349a7cc","Type":"ContainerStarted","Data":"1bf31569b29eaa967754529bab060ef16dc67f490f8903d46b73b29628056e28"} Oct 30 21:41:09.704439 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.704311 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" event={"ID":"b99efd6a-6f7d-444a-89dd-bf292349a7cc","Type":"ContainerStarted","Data":"320927bfb46c5b02dd5f0df6648d8b1171cdcbdde4bee7fd24000f6132bd70cc"} Oct 30 21:41:09.706961 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.706937 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" event={"ID":"9e3895a8-b605-453e-8b79-93b54f506dfb","Type":"ContainerStarted","Data":"6b0b8f133c1075b48c33ad18505557780d189c2034c8e5824b37eafbbe385ebc"} Oct 30 21:41:09.707025 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.706988 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" event={"ID":"9e3895a8-b605-453e-8b79-93b54f506dfb","Type":"ContainerStarted","Data":"ae3cf7a868203a39f8bb0ece30492cfe0835e8672bc3db16fbbdc4effad108c5"} Oct 30 21:41:09.709727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.709706 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" event={"ID":"c1ee1395-263a-4afa-8f55-244a8edcd3a8","Type":"ContainerStarted","Data":"c136f0caf0208ce9494c36ce7c024a1280db31ec7ff6130258c7bde4b55e827a"} Oct 30 21:41:09.709985 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.709924 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" event={"ID":"c1ee1395-263a-4afa-8f55-244a8edcd3a8","Type":"ContainerStarted","Data":"aed51a9989fb7a02f03c8e420c8485cf2328ee4d8013d560b4a4edd358b13fd8"} Oct 30 21:41:09.712681 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.712203 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" event={"ID":"8d851d22-81c2-4ba2-90b7-e5d82d1b0929","Type":"ContainerStarted","Data":"12eeaf9980be6d48efb4f17b3c2419661a59fdf0cb17830db2d63f150605f466"} Oct 30 21:41:09.712681 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.712248 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" event={"ID":"8d851d22-81c2-4ba2-90b7-e5d82d1b0929","Type":"ContainerStarted","Data":"8c3e92e8fe913414284fb24c63c906a2edc8656af6cc42e3f7b29bacc35e924e"} Oct 30 21:41:09.736925 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.736862 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" podStartSLOduration=1.7368282769999999 podStartE2EDuration="1.736828277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:09.733296726 +0000 UTC m=+738.329113413" watchObservedRunningTime="2026-10-30 21:41:09.736828277 +0000 UTC m=+738.332644938" Oct 30 21:41:09.800826 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.793079 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" podStartSLOduration=1.793055786 podStartE2EDuration="1.793055786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:09.763039048 +0000 UTC m=+738.358855748" watchObservedRunningTime="2026-10-30 21:41:09.793055786 +0000 UTC m=+738.388872448" Oct 30 21:41:09.812696 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.812633 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" podStartSLOduration=1.8126151 podStartE2EDuration="1.8126151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:09.80501619 +0000 UTC m=+738.400832871" watchObservedRunningTime="2026-10-30 21:41:09.8126151 +0000 UTC m=+738.408431761" Oct 30 21:41:09.848176 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.835615 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" podStartSLOduration=1.835592338 podStartE2EDuration="1.835592338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:09.83031021 +0000 UTC m=+738.426126872" watchObservedRunningTime="2026-10-30 21:41:09.835592338 +0000 UTC m=+738.431408999" Oct 30 21:41:09.848176 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.838946 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:09.848176 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:09.841238 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:10.050786 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:10.050428 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2eda136-jh2tv" Oct 30 21:41:10.140030 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:10.139390 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:10.152693 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:10.152655 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:41:10.164214 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:10.164172 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" podUID="b99efd6a-6f7d-444a-89dd-bf292349a7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 21:41:10.164711 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:10.163659 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:10.259865 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:10.247740 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7836e324-455e-47b4-96df-586ef9cfd53e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7836e324-455e-47b4-96df-586ef9cfd53e-maas-api-openshift-ca\") pod \"7836e324-455e-47b4-96df-586ef9cfd53e\" (UID: \"7836e324-455e-47b4-96df-586ef9cfd53e\") " Oct 30 21:41:10.259865 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:10.259390 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7836e324-455e-47b4-96df-586ef9cfd53e-kube-api-access-vjlqx\" (UniqueName: \"kubernetes.io/projected/7836e324-455e-47b4-96df-586ef9cfd53e-kube-api-access-vjlqx\") pod \"7836e324-455e-47b4-96df-586ef9cfd53e\" (UID: \"7836e324-455e-47b4-96df-586ef9cfd53e\") " Oct 30 21:41:10.259865 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:10.259461 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7836e324-455e-47b4-96df-586ef9cfd53e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7836e324-455e-47b4-96df-586ef9cfd53e-maas-api-xks-ca\") pod \"7836e324-455e-47b4-96df-586ef9cfd53e\" (UID: \"7836e324-455e-47b4-96df-586ef9cfd53e\") " Oct 30 21:41:10.261255 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:10.257977 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7836e324-455e-47b4-96df-586ef9cfd53e-maas-api-openshift-ca" pod "7836e324-455e-47b4-96df-586ef9cfd53e" (UID: "7836e324-455e-47b4-96df-586ef9cfd53e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:10.305083 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:10.304982 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7836e324-455e-47b4-96df-586ef9cfd53e-maas-api-xks-ca" pod "7836e324-455e-47b4-96df-586ef9cfd53e" (UID: "7836e324-455e-47b4-96df-586ef9cfd53e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:10.308781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:10.305871 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7836e324-455e-47b4-96df-586ef9cfd53e-kube-api-access-vjlqx" pod "7836e324-455e-47b4-96df-586ef9cfd53e" (UID: "7836e324-455e-47b4-96df-586ef9cfd53e"). InnerVolumeSpecName "kube-api-access-vjlqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:10.379777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:10.372402 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7836e324-455e-47b4-96df-586ef9cfd53e-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:10.379777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:10.372445 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjlqx\" (UniqueName: \"kubernetes.io/projected/7836e324-455e-47b4-96df-586ef9cfd53e-kube-api-access-vjlqx\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:10.379777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:10.372461 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7836e324-455e-47b4-96df-586ef9cfd53e-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:10.674978 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:10.674940 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:10.675381 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:10.675359 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:41:10.675519 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:10.675499 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" podUID="9e3895a8-b605-453e-8b79-93b54f506dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 21:41:10.738791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:10.738670 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2eda136-jh2tv" Oct 30 21:41:10.739901 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:10.739807 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2eda136-jh2tv" event={"ID":"7836e324-455e-47b4-96df-586ef9cfd53e","Type":"ContainerDied","Data":"f415ac6ecc2c5f1f53cc257d929028869b886cc767f9f13982378ba7970acd4b"} Oct 30 21:41:10.739901 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:10.739869 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f415ac6ecc2c5f1f53cc257d929028869b886cc767f9f13982378ba7970acd4b" Oct 30 21:41:10.849307 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:10.849271 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:41:10.849562 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:10.849531 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" podUID="8d851d22-81c2-4ba2-90b7-e5d82d1b0929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 21:41:10.849990 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:10.849966 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:41:10.850326 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:10.850286 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" podUID="c1ee1395-263a-4afa-8f55-244a8edcd3a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 21:41:11.150774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:11.149912 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:41:11.150774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:11.149981 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" podUID="b99efd6a-6f7d-444a-89dd-bf292349a7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 21:41:11.683780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:11.683007 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:41:11.683780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:11.683067 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" podUID="9e3895a8-b605-453e-8b79-93b54f506dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 21:41:11.718775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:11.718030 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33edf72f-ddl6r"] Oct 30 21:41:11.718959 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:11.718939 5801 cpu_manager.go:414] "containerMap: removing container" podUID="7836e324-455e-47b4-96df-586ef9cfd53e" containerName="revoke-keys" Oct 30 21:41:11.719021 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:11.718958 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7836e324-455e-47b4-96df-586ef9cfd53e" containerName="revoke-keys" Oct 30 21:41:11.733777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:11.719181 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="7836e324-455e-47b4-96df-586ef9cfd53e" containerName="revoke-keys" Oct 30 21:41:11.758456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:11.744995 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33edf72f-ddl6r" Oct 30 21:41:11.801131 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:11.801032 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:41:11.801131 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:11.801120 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:41:11.801265 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:11.801221 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:41:11.802112 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:11.801376 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:41:11.844777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:11.843017 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:41:11.844777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:11.843078 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" podUID="8d851d22-81c2-4ba2-90b7-e5d82d1b0929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 21:41:11.844777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:11.843653 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:41:11.844777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:11.843696 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" podUID="c1ee1395-263a-4afa-8f55-244a8edcd3a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 21:41:11.877776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:11.857862 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/314732a1-75c0-4928-8467-9a3bc84729ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33edf72f-ddl6r\" (UID: \"314732a1-75c0-4928-8467-9a3bc84729ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33edf72f-ddl6r" Oct 30 21:41:11.877776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:11.857936 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/314732a1-75c0-4928-8467-9a3bc84729ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33edf72f-ddl6r\" (UID: \"314732a1-75c0-4928-8467-9a3bc84729ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33edf72f-ddl6r" Oct 30 21:41:11.877776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:11.858071 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59rdx\" (UniqueName: \"kubernetes.io/projected/314732a1-75c0-4928-8467-9a3bc84729ff-kube-api-access-59rdx\") pod \"maas-api-revoke-sub-33edf72f-ddl6r\" (UID: \"314732a1-75c0-4928-8467-9a3bc84729ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33edf72f-ddl6r" Oct 30 21:41:11.982840 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:11.981931 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/314732a1-75c0-4928-8467-9a3bc84729ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33edf72f-ddl6r\" (UID: \"314732a1-75c0-4928-8467-9a3bc84729ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33edf72f-ddl6r" Oct 30 21:41:11.982840 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:11.982043 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/314732a1-75c0-4928-8467-9a3bc84729ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33edf72f-ddl6r\" (UID: \"314732a1-75c0-4928-8467-9a3bc84729ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33edf72f-ddl6r" Oct 30 21:41:11.982840 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:11.982368 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59rdx\" (UniqueName: \"kubernetes.io/projected/314732a1-75c0-4928-8467-9a3bc84729ff-kube-api-access-59rdx\") pod \"maas-api-revoke-sub-33edf72f-ddl6r\" (UID: \"314732a1-75c0-4928-8467-9a3bc84729ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33edf72f-ddl6r" Oct 30 21:41:12.016831 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.005238 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:12.019910 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.018823 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/314732a1-75c0-4928-8467-9a3bc84729ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33edf72f-ddl6r\" (UID: \"314732a1-75c0-4928-8467-9a3bc84729ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33edf72f-ddl6r" Oct 30 21:41:12.021193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.021167 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/314732a1-75c0-4928-8467-9a3bc84729ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33edf72f-ddl6r\" (UID: \"314732a1-75c0-4928-8467-9a3bc84729ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33edf72f-ddl6r" Oct 30 21:41:12.061359 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.061323 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33edf72f-ddl6r"] Oct 30 21:41:12.149960 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.149903 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:12.151802 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.151767 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:12.193775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.185898 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:41:12.193775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.185967 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" podUID="b99efd6a-6f7d-444a-89dd-bf292349a7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 21:41:12.193775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.190675 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59rdx\" (UniqueName: \"kubernetes.io/projected/314732a1-75c0-4928-8467-9a3bc84729ff-kube-api-access-59rdx\") pod \"maas-api-revoke-sub-33edf72f-ddl6r\" (UID: \"314732a1-75c0-4928-8467-9a3bc84729ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33edf72f-ddl6r" Oct 30 21:41:12.245831 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.237192 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:12.245831 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.242680 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:12.284178 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.284143 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:12.373490 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.373446 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:12.403962 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.403918 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:12.404638 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.404205 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:12.409264 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.404234 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:41:12.492779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.487999 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33edf72f-ddl6r" Oct 30 21:41:12.518825 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.506494 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:41:12.518825 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.506893 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:41:12.568780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.567682 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:12.605773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.605485 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": context deadline exceeded" Oct 30 21:41:12.704890 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.704851 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:41:12.705131 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.705103 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" podUID="9e3895a8-b605-453e-8b79-93b54f506dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 21:41:12.840778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.836284 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:41:12.840778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.836346 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" podUID="8d851d22-81c2-4ba2-90b7-e5d82d1b0929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 21:41:12.850000 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.842454 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:41:12.850000 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.842520 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" podUID="c1ee1395-263a-4afa-8f55-244a8edcd3a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 21:41:12.995778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:12.988056 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:13.018776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.008301 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:13.018776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.008694 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:41:13.249354 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.177849 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:13.249354 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.178639 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:41:13.249354 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.203869 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:41:13.249354 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.204332 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:41:13.249354 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.208515 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:41:13.249354 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.208568 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" podUID="b99efd6a-6f7d-444a-89dd-bf292349a7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 21:41:13.249354 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.245880 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:41:13.249354 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.246316 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:41:13.397896 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.397845 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:13.403265 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.400936 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:41:13.429962 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.413631 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:13.429962 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.414693 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:41:13.429962 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.424782 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:13.429962 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.425095 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:41:13.547907 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.434810 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:41:13.547907 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.462909 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:41:13.826238 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.796224 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:41:13.826238 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.796441 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:41:13.826238 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.797573 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:41:13.826238 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.797779 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:41:13.971784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.953355 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:41:13.997914 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.997864 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" podUID="9e3895a8-b605-453e-8b79-93b54f506dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 21:41:14.134837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:13.976872 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:14.134837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:14.045947 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:14.186779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:14.166615 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:41:14.186779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:14.166766 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:41:14.186779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:14.166808 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:41:14.224779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:14.219063 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:41:14.224779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:14.219124 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" podUID="c1ee1395-263a-4afa-8f55-244a8edcd3a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 21:41:14.299130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:14.232884 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:41:14.299130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:14.232948 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" podUID="8d851d22-81c2-4ba2-90b7-e5d82d1b0929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 21:41:14.303146 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:14.303109 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:14.404836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:14.340286 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:41:14.404836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:14.340349 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" podUID="b99efd6a-6f7d-444a-89dd-bf292349a7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 21:41:14.404836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:14.344428 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:41:14.404836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:14.357463 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:41:14.441784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:14.416382 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:14.441784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:14.417692 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:14.441784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:14.434545 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:14.462447 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:14.462404 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:41:14.908778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:14.881151 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:41:14.908778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:14.881233 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" podUID="9e3895a8-b605-453e-8b79-93b54f506dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 21:41:15.040846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:14.985706 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:41:15.040846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:14.986110 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:41:15.040846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.021914 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:15.111813 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.104907 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:41:15.111813 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.105336 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:41:15.111813 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.106600 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:41:15.111813 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.106638 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" podUID="c1ee1395-263a-4afa-8f55-244a8edcd3a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 21:41:15.111813 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.108253 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:15.124784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.124405 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:15.154843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.154804 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:41:15.183833 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.158869 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:41:15.243835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.239943 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:41:15.521308 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.521205 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" podUID="b99efd6a-6f7d-444a-89dd-bf292349a7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 21:41:15.521892 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.359016 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:41:15.790592 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.381059 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:41:15.790844 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.467152 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 21:41:15.905481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.467484 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:41:15.905481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.515126 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:41:15.905481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.515162 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:41:15.905481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.347148 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:15.905481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.548351 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:15.905481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.671727 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:41:15.905481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.837526 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" podUID="8d851d22-81c2-4ba2-90b7-e5d82d1b0929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 21:41:15.905481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.679780 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:41:15.905481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.837592 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" podUID="9e3895a8-b605-453e-8b79-93b54f506dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 21:41:15.905481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.679857 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded" Oct 30 21:41:15.946926 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:15.921937 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:16.042282 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.003262 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:41:16.042282 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.003339 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" podUID="c1ee1395-263a-4afa-8f55-244a8edcd3a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 21:41:16.042282 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.003455 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:41:16.042282 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.003480 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" podUID="8d851d22-81c2-4ba2-90b7-e5d82d1b0929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 21:41:16.042282 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.026310 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:16.042282 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.030560 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:41:16.042282 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.042004 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:16.136789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.107920 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:41:16.136789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.108542 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:16.136789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.108954 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:16.226913 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.226863 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:16.230489 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.230441 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:16.285650 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.285313 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:41:16.285650 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.285375 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" podUID="b99efd6a-6f7d-444a-89dd-bf292349a7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 21:41:16.532442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.532415 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:41:16.533281 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.533245 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33edf72f-ddl6r" event={"ID":"314732a1-75c0-4928-8467-9a3bc84729ff","Type":"ContainerStarted","Data":"3577d973b74861b30fb662ecc78f0ca43ad15ffbf887537e2e45f31262a6305c"} Oct 30 21:41:16.551937 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.551910 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:41:16.552364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.552251 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:41:16.552553 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.552537 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33edf72f-ddl6r"] Oct 30 21:41:16.653787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.650065 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:41:16.702218 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.702086 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:41:16.702218 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.702145 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" podUID="9e3895a8-b605-453e-8b79-93b54f506dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 21:41:16.798358 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.796008 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:16.798358 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.796654 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:16.798358 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.797830 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:16.810872 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.809921 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:16.830825 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.828841 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:16.837970 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.837927 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:16.838238 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.838209 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:41:16.838316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.838259 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" podUID="8d851d22-81c2-4ba2-90b7-e5d82d1b0929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 21:41:16.838316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.837932 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:16.839463 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.839443 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:41:16.870773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.870588 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:41:16.870773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:16.870659 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" podUID="c1ee1395-263a-4afa-8f55-244a8edcd3a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 21:41:17.026023 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:17.023593 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:41:17.039671 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:17.039644 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:41:17.041225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:17.041206 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:41:17.050260 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:17.050241 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:41:17.110838 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:17.110419 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:41:17.135322 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:17.135286 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:41:17.135703 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:17.135671 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" podUID="b99efd6a-6f7d-444a-89dd-bf292349a7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 21:41:17.136214 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:17.136113 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:41:17.148007 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:17.147119 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:41:17.151061 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:17.149808 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:41:17.151061 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:17.150520 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:41:17.451260 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:17.451225 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33edf72f-ddl6r" event={"ID":"314732a1-75c0-4928-8467-9a3bc84729ff","Type":"ContainerStarted","Data":"5c00278fea947c0b190826252d0afbf34c26d256bb5ea7ad430b8a532c4be2d1"} Oct 30 21:41:17.638412 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:17.638244 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33edf72f-ddl6r" podStartSLOduration=6.638222392 podStartE2EDuration="6.638222392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:17.502065815 +0000 UTC m=+746.097882500" watchObservedRunningTime="2026-10-30 21:41:17.638222392 +0000 UTC m=+746.234039053" Oct 30 21:41:17.874794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:17.874027 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:41:17.874794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:17.874116 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" podUID="8d851d22-81c2-4ba2-90b7-e5d82d1b0929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 21:41:18.138453 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:18.138414 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:41:18.138784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:18.138736 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" podUID="b99efd6a-6f7d-444a-89dd-bf292349a7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 21:41:18.325773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:18.324298 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:18.336813 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:18.330086 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:18.344812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:18.343250 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:41:18.476904 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:18.473661 5801 generic.go:356] "Generic (PLEG): container finished" podID="314732a1-75c0-4928-8467-9a3bc84729ff" containerID="5c00278fea947c0b190826252d0afbf34c26d256bb5ea7ad430b8a532c4be2d1" exitCode=0 Oct 30 21:41:18.476904 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:18.474609 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33edf72f-ddl6r" event={"ID":"314732a1-75c0-4928-8467-9a3bc84729ff","Type":"ContainerDied","Data":"5c00278fea947c0b190826252d0afbf34c26d256bb5ea7ad430b8a532c4be2d1"} Oct 30 21:41:18.572128 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:18.572096 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:18.573026 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:18.572994 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:18.575396 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:18.575372 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:41:18.840305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:18.840263 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:18.840707 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:18.840618 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:18.841462 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:18.841437 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:41:18.933310 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:18.933279 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:19.317807 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:19.314747 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:19.317807 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:19.315899 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:19.317807 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:19.317287 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:41:19.648140 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:19.648115 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33edf72f-ddl6r" Oct 30 21:41:19.771906 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:19.771872 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/314732a1-75c0-4928-8467-9a3bc84729ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/314732a1-75c0-4928-8467-9a3bc84729ff-maas-api-openshift-ca\") pod \"314732a1-75c0-4928-8467-9a3bc84729ff\" (UID: \"314732a1-75c0-4928-8467-9a3bc84729ff\") " Oct 30 21:41:19.772054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:19.771957 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/314732a1-75c0-4928-8467-9a3bc84729ff-kube-api-access-59rdx\" (UniqueName: \"kubernetes.io/projected/314732a1-75c0-4928-8467-9a3bc84729ff-kube-api-access-59rdx\") pod \"314732a1-75c0-4928-8467-9a3bc84729ff\" (UID: \"314732a1-75c0-4928-8467-9a3bc84729ff\") " Oct 30 21:41:19.772054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:19.772050 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/314732a1-75c0-4928-8467-9a3bc84729ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/314732a1-75c0-4928-8467-9a3bc84729ff-maas-api-xks-ca\") pod \"314732a1-75c0-4928-8467-9a3bc84729ff\" (UID: \"314732a1-75c0-4928-8467-9a3bc84729ff\") " Oct 30 21:41:19.772281 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:19.772253 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/314732a1-75c0-4928-8467-9a3bc84729ff-maas-api-openshift-ca" pod "314732a1-75c0-4928-8467-9a3bc84729ff" (UID: "314732a1-75c0-4928-8467-9a3bc84729ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:19.772525 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:19.772498 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/314732a1-75c0-4928-8467-9a3bc84729ff-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:19.774027 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:19.773986 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/314732a1-75c0-4928-8467-9a3bc84729ff-maas-api-xks-ca" pod "314732a1-75c0-4928-8467-9a3bc84729ff" (UID: "314732a1-75c0-4928-8467-9a3bc84729ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:19.774143 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:19.774091 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/314732a1-75c0-4928-8467-9a3bc84729ff-kube-api-access-59rdx" pod "314732a1-75c0-4928-8467-9a3bc84729ff" (UID: "314732a1-75c0-4928-8467-9a3bc84729ff"). InnerVolumeSpecName "kube-api-access-59rdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:19.874660 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:19.874568 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59rdx\" (UniqueName: \"kubernetes.io/projected/314732a1-75c0-4928-8467-9a3bc84729ff-kube-api-access-59rdx\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:19.874660 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:19.874607 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/314732a1-75c0-4928-8467-9a3bc84729ff-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:20.270348 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:20.270231 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:20.501894 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:20.501866 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33edf72f-ddl6r" Oct 30 21:41:20.502046 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:20.501911 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33edf72f-ddl6r" event={"ID":"314732a1-75c0-4928-8467-9a3bc84729ff","Type":"ContainerDied","Data":"3577d973b74861b30fb662ecc78f0ca43ad15ffbf887537e2e45f31262a6305c"} Oct 30 21:41:20.502046 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:20.501959 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3577d973b74861b30fb662ecc78f0ca43ad15ffbf887537e2e45f31262a6305c" Oct 30 21:41:21.391208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:21.391171 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:21.533321 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:21.533287 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1caf5a9b-ndfhz"] Oct 30 21:41:21.533877 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:21.533862 5801 cpu_manager.go:414] "containerMap: removing container" podUID="314732a1-75c0-4928-8467-9a3bc84729ff" containerName="revoke-keys" Oct 30 21:41:21.533938 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:21.533878 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="314732a1-75c0-4928-8467-9a3bc84729ff" containerName="revoke-keys" Oct 30 21:41:21.534018 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:21.534009 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="314732a1-75c0-4928-8467-9a3bc84729ff" containerName="revoke-keys" Oct 30 21:41:21.537767 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:21.537728 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1caf5a9b-ndfhz" Oct 30 21:41:21.545381 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:21.545184 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:41:21.546038 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:21.545569 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:41:21.546038 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:21.545608 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:41:21.546038 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:21.545700 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:41:21.550130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:21.549966 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1caf5a9b-ndfhz"] Oct 30 21:41:21.597487 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:21.597453 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqcsd\" (UniqueName: \"kubernetes.io/projected/b1485533-fd44-4182-a51a-8ed758b41442-kube-api-access-kqcsd\") pod \"maas-api-revoke-sub-1caf5a9b-ndfhz\" (UID: \"b1485533-fd44-4182-a51a-8ed758b41442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1caf5a9b-ndfhz" Oct 30 21:41:21.597661 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:21.597637 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1485533-fd44-4182-a51a-8ed758b41442-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1caf5a9b-ndfhz\" (UID: \"b1485533-fd44-4182-a51a-8ed758b41442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1caf5a9b-ndfhz" Oct 30 21:41:21.597841 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:21.597821 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1485533-fd44-4182-a51a-8ed758b41442-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1caf5a9b-ndfhz\" (UID: \"b1485533-fd44-4182-a51a-8ed758b41442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1caf5a9b-ndfhz" Oct 30 21:41:21.699423 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:21.699343 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1485533-fd44-4182-a51a-8ed758b41442-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1caf5a9b-ndfhz\" (UID: \"b1485533-fd44-4182-a51a-8ed758b41442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1caf5a9b-ndfhz" Oct 30 21:41:21.699589 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:21.699448 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1485533-fd44-4182-a51a-8ed758b41442-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1caf5a9b-ndfhz\" (UID: \"b1485533-fd44-4182-a51a-8ed758b41442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1caf5a9b-ndfhz" Oct 30 21:41:21.699589 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:21.699567 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqcsd\" (UniqueName: \"kubernetes.io/projected/b1485533-fd44-4182-a51a-8ed758b41442-kube-api-access-kqcsd\") pod \"maas-api-revoke-sub-1caf5a9b-ndfhz\" (UID: \"b1485533-fd44-4182-a51a-8ed758b41442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1caf5a9b-ndfhz" Oct 30 21:41:21.700183 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:21.700160 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1485533-fd44-4182-a51a-8ed758b41442-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1caf5a9b-ndfhz\" (UID: \"b1485533-fd44-4182-a51a-8ed758b41442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1caf5a9b-ndfhz" Oct 30 21:41:21.701805 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:21.701786 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1485533-fd44-4182-a51a-8ed758b41442-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1caf5a9b-ndfhz\" (UID: \"b1485533-fd44-4182-a51a-8ed758b41442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1caf5a9b-ndfhz" Oct 30 21:41:21.715590 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:21.715566 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqcsd\" (UniqueName: \"kubernetes.io/projected/b1485533-fd44-4182-a51a-8ed758b41442-kube-api-access-kqcsd\") pod \"maas-api-revoke-sub-1caf5a9b-ndfhz\" (UID: \"b1485533-fd44-4182-a51a-8ed758b41442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1caf5a9b-ndfhz" Oct 30 21:41:21.860110 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:21.860074 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1caf5a9b-ndfhz" Oct 30 21:41:22.148942 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:41:22.148908 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1485533_fd44_4182_a51a_8ed758b41442.slice/crio-70df591428bd955be79ef960ccae579bddc7fa2ae2f6d15b0bf35e1a553eb229 WatchSource:0}: Error finding container 70df591428bd955be79ef960ccae579bddc7fa2ae2f6d15b0bf35e1a553eb229: Status 404 returned error can't find the container with id 70df591428bd955be79ef960ccae579bddc7fa2ae2f6d15b0bf35e1a553eb229 Oct 30 21:41:22.152260 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:22.152237 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1caf5a9b-ndfhz"] Oct 30 21:41:22.411522 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:22.411435 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:22.537457 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:22.537405 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1caf5a9b-ndfhz" event={"ID":"b1485533-fd44-4182-a51a-8ed758b41442","Type":"ContainerStarted","Data":"70df591428bd955be79ef960ccae579bddc7fa2ae2f6d15b0bf35e1a553eb229"} Oct 30 21:41:23.471786 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:23.471562 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42"] Oct 30 21:41:23.792835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:23.732708 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:23.948781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:23.883628 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42"] Oct 30 21:41:24.293790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:23.945315 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cc8b5b-1443-40da-a41d-745fcba8e474-istiod-ca-cert\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.293790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:23.976974 5801 generic.go:356] "Generic (PLEG): container finished" podID="b1485533-fd44-4182-a51a-8ed758b41442" containerID="60058831d0e778e3919cd2bfa9160fb72e3a15b4658c5dc371655a08aaccb024" exitCode=0 Oct 30 21:41:24.293790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.047132 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:41:24.293790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.118974 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:41:24.293790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.119146 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:41:24.294180 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.119459 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:24.294180 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.294067 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k"] Oct 30 21:41:24.332791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.119701 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:24.332791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.325941 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-credential-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.332791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.329892 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-envoy\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.332791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.329948 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-workload-certs\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.362003 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.361958 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:24.363408 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.329990 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d44bq\" (UniqueName: \"kubernetes.io/projected/b8cc8b5b-1443-40da-a41d-745fcba8e474-kube-api-access-d44bq\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.363658 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.363624 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-podinfo\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.363858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.363837 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-data\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.364102 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.364083 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-workload-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.364300 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.364284 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-token\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.365812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.364683 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:24.372977 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.372953 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k"] Oct 30 21:41:24.373138 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.373119 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1caf5a9b-ndfhz" event={"ID":"b1485533-fd44-4182-a51a-8ed758b41442","Type":"ContainerDied","Data":"60058831d0e778e3919cd2bfa9160fb72e3a15b4658c5dc371655a08aaccb024"} Oct 30 21:41:24.373250 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.373141 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:24.393287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.373045 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:24.440549 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.440502 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:24.483966 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.483916 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:24.593598 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.581043 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:41:24.834781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.833015 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:41:24.860906 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.852236 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-envoy\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.861174 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.861144 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-workload-certs\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.861337 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.861317 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d44bq\" (UniqueName: \"kubernetes.io/projected/b8cc8b5b-1443-40da-a41d-745fcba8e474-kube-api-access-d44bq\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.861458 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.861442 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-podinfo\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.861597 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.861566 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:24.861706 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.861692 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-data\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.861888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.861867 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-workload-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.862043 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.862028 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-token\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.865775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.862256 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cc8b5b-1443-40da-a41d-745fcba8e474-istiod-ca-cert\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.865983 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.865960 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:24.866143 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.866124 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-credential-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.885161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.885110 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:24.885772 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.870147 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-credential-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.885948 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.870721 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-workload-certs\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.887235 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.887211 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-workload-socket\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.909864 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.887884 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-data\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.910122 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.888494 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cc8b5b-1443-40da-a41d-745fcba8e474-istiod-ca-cert\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.937777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.936597 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-envoy\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.950784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.938571 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d44bq\" (UniqueName: \"kubernetes.io/projected/b8cc8b5b-1443-40da-a41d-745fcba8e474-kube-api-access-d44bq\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.963778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.956416 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-token\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:24.981780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.964433 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-podinfo\") pod \"e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:25.002775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.997441 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:25.002775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.997546 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dr6r\" (UniqueName: \"kubernetes.io/projected/528a2150-a9c8-42d4-8203-113bc618287a-kube-api-access-6dr6r\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:25.002775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.997596 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:25.002775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.997637 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528a2150-a9c8-42d4-8203-113bc618287a-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:25.002775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:24.998238 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:25.003318 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.003287 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/528a2150-a9c8-42d4-8203-113bc618287a-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:25.003481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.003463 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:25.003735 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.003715 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:25.004259 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.004239 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:25.004448 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.004431 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528a2150-a9c8-42d4-8203-113bc618287a-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:25.004714 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.004161 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:25.035931 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.035871 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:25.053087 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.053048 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z"] Oct 30 21:41:25.118843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.094900 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:25.121523 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.121483 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528a2150-a9c8-42d4-8203-113bc618287a-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:25.197778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.191958 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:25.197778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.192067 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dr6r\" (UniqueName: \"kubernetes.io/projected/528a2150-a9c8-42d4-8203-113bc618287a-kube-api-access-6dr6r\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:25.197778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.192115 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528a2150-a9c8-42d4-8203-113bc618287a-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:25.197778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.192290 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/528a2150-a9c8-42d4-8203-113bc618287a-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:25.197778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.192345 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:25.197778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.192475 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:25.197778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.167084 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:25.197778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.194606 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:25.197778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.168099 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528a2150-a9c8-42d4-8203-113bc618287a-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:25.237719 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.217337 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528a2150-a9c8-42d4-8203-113bc618287a-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:25.237719 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.217636 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:25.237719 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.221702 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:25.278808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.277807 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:25.298541 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.298394 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:25.299677 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.299646 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:41:25.322788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.320576 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:41:25.349337 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.349299 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:41:25.349645 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.349321 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv98j\" (UniqueName: \"kubernetes.io/projected/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-kube-api-access-rv98j\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:25.349900 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.349879 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-workload-certs\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:25.350076 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.350058 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-token\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:25.350244 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.350225 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-credential-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:25.350381 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.350365 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istiod-ca-cert\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:25.350528 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.350505 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-envoy\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:25.350694 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.350677 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-data\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:25.350888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.350871 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-workload-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:25.351021 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.351005 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-podinfo\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:25.365156 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.365115 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:25.375820 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.375703 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:41:25.376095 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.376064 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dr6r\" (UniqueName: \"kubernetes.io/projected/528a2150-a9c8-42d4-8203-113bc618287a-kube-api-access-6dr6r\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:25.434037 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.433982 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:25.484785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.479897 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:25.524587 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.524529 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:41:25.674763 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.674710 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/528a2150-a9c8-42d4-8203-113bc618287a-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:25.674998 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.577267 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-workload-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:25.675157 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.587181 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:25.675685 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.588009 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 21:41:25.676121 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.588038 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:41:25.676470 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.588074 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": context deadline exceeded" Oct 30 21:41:25.676602 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.623565 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:41:25.683885 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.576610 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-workload-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:25.684142 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.684114 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-podinfo\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:25.684535 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.684513 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv98j\" (UniqueName: \"kubernetes.io/projected/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-kube-api-access-rv98j\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:25.687976 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.687950 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-workload-certs\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:25.688172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.688152 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-token\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:25.688323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.688306 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-credential-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:25.688447 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.688430 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istiod-ca-cert\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:25.692773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.688590 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-envoy\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:25.693013 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.692990 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-data\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:25.693704 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.693681 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-data\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:25.712333 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.712297 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istiod-ca-cert\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:26.142905 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.739308 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-workload-certs\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:26.143175 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.739529 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-credential-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:26.143283 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.775791 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:26.143380 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.809121 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-envoy\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:26.143470 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.844585 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:26.156628 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.979591 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv98j\" (UniqueName: \"kubernetes.io/projected/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-kube-api-access-rv98j\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:26.156882 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.980117 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-podinfo\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:26.799779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:25.995740 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:41:26.803777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:26.800485 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:41:26.804051 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:26.804026 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:41:26.807872 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:26.807842 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z"] Oct 30 21:41:26.808110 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:26.808094 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:41:26.836891 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:26.836855 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:41:26.837138 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:26.837122 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:41:26.837267 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:26.837255 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:41:26.837559 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:26.348957 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:26.847781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:26.413711 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:26.880778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:26.636969 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:41:26.880778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:26.637028 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:41:26.880778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:26.637052 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:41:26.880778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:26.224141 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:26.880778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:26.230101 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded" Oct 30 21:41:26.880778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:26.319508 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:41:26.880778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:26.680651 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:26.880778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:26.046973 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:26.880778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:26.832853 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:26.880778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:26.875924 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:41:26.880778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:26.875984 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:41:26.922935 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:26.922893 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:41:26.940959 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:26.927167 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 21:41:26.941560 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:26.928185 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:41:27.042243 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:27.042205 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-token\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:27.178942 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:27.157975 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:41:27.503836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:27.472743 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:27.739021 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:27.738876 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4"] Oct 30 21:41:27.853777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:27.840877 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:41:27.880184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:27.880141 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:27.883887 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:27.880457 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:27.946446 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:27.880490 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:27.947048 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:27.883873 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4"] Oct 30 21:41:27.947234 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:27.885811 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:41:27.955664 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:27.885844 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:41:27.957905 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:27.957861 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:27.969777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:27.958140 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 21:41:27.969777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:27.958257 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:41:27.969777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:27.920899 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:27.969777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:27.931707 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:41:27.969777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:27.920977 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:41:27.969777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:27.920935 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:41:27.969777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:27.920957 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:41:28.015822 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:27.999467 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-credential-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.016120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.016083 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-token\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.016464 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.016442 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrpwb\" (UniqueName: \"kubernetes.io/projected/6aaf74ef-1550-443a-a046-d08fc45e07f1-kube-api-access-rrpwb\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.016607 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.016590 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-envoy\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.016728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.016712 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-podinfo\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.016858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.016841 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-data\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.020775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.016964 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-workload-certs\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.024775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.021182 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-workload-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.045240 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.045197 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aaf74ef-1550-443a-a046-d08fc45e07f1-istiod-ca-cert\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.148079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.148034 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-workload-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.173798 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.173736 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aaf74ef-1550-443a-a046-d08fc45e07f1-istiod-ca-cert\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.174949 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.174920 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-credential-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.175148 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.175125 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-token\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.175428 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.175405 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrpwb\" (UniqueName: \"kubernetes.io/projected/6aaf74ef-1550-443a-a046-d08fc45e07f1-kube-api-access-rrpwb\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.175592 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.175573 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-envoy\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.175732 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.175714 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-podinfo\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.175884 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.175865 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-data\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.176001 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.175987 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-workload-certs\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.176704 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.176682 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-workload-certs\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.198858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.158703 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-workload-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.199074 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.177410 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-credential-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.211995 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.174808 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aaf74ef-1550-443a-a046-d08fc45e07f1-istiod-ca-cert\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.212205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.190124 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-data\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.212314 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.195732 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-podinfo\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.495839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.240850 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:41:28.495839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.373675 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrpwb\" (UniqueName: \"kubernetes.io/projected/6aaf74ef-1550-443a-a046-d08fc45e07f1-kube-api-access-rrpwb\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.495839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.421423 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-token\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.495839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.465857 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sk2kw" podUID="faea70fa-2fa1-45b8-9409-72fb5d589cc3" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.25:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:28.624777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.577826 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": context deadline exceeded" Oct 30 21:41:28.648871 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.648817 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:28.853965 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.814252 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-envoy\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.853965 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.814408 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:28.853965 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.841108 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:28.877171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.877124 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:28.912045 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.911324 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:41:28.984835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.963863 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:28.984835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.964330 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:41:28.984835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.964776 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:28.984835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.965054 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:41:28.984835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.978876 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:41:28.984835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.979296 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:41:28.984835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.979628 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:28.992910 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.992862 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:41:28.999790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:28.993505 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:41:29.011937 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:29.011866 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:29.294087 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:29.294041 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:29.579375 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:29.579324 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:29.784841 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:29.634321 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:29.943114 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:29.888912 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:29.943114 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:29.917552 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:41:30.212179 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:30.193692 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:41:30.213620 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:30.195838 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 21:41:30.301831 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:30.298021 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:30.342779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:30.328285 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:41:30.345060 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:30.345012 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:41:30.346727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:30.346680 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:41:30.432964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:30.432924 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:41:30.451986 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:30.451924 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:41:30.514774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:30.452262 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:30.514774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:30.452335 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:41:30.962839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:30.883052 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:41:31.186933 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:31.186887 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:41:31.496073 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:31.187173 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:41:31.496597 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:31.280887 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:31.847795 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:31.847736 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:31.847919 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:31.383619 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:31.848152 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:31.383690 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:41:31.848892 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:31.600688 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:41:31.849305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:31.624746 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:41:31.849678 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:31.778323 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded" Oct 30 21:41:31.849841 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:31.781042 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:41:31.858183 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:31.781158 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:41:31.858550 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:31.781188 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:31.863782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:31.794588 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:41:31.897014 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:31.794624 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:41:31.897014 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:31.322414 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1caf5a9b-ndfhz" Oct 30 21:41:31.899998 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:31.899959 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:31.978829 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:31.978795 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:31.979368 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:31.979343 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:31.979502 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:31.979485 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:31.989339 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:31.981212 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:31.989339 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:31.981286 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:31.989339 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:31.981329 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:32.023146 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:41:32.015935 5801 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.322s" Oct 30 21:41:32.023146 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.015976 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" event={"ID":"b8cc8b5b-1443-40da-a41d-745fcba8e474","Type":"ContainerStarted","Data":"d8004bcacd9d4807d9a7f69c6fcc44ef63731f9384c38c610a7e0ff8d6c192d3"} Oct 30 21:41:32.023146 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.016025 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1caf5a9b-ndfhz" event={"ID":"b1485533-fd44-4182-a51a-8ed758b41442","Type":"ContainerDied","Data":"70df591428bd955be79ef960ccae579bddc7fa2ae2f6d15b0bf35e1a553eb229"} Oct 30 21:41:32.023146 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.016053 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70df591428bd955be79ef960ccae579bddc7fa2ae2f6d15b0bf35e1a553eb229" Oct 30 21:41:32.047424 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:41:32.047389 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:41:32.064231 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.064136 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1485533-fd44-4182-a51a-8ed758b41442-kube-api-access-kqcsd\" (UniqueName: \"kubernetes.io/projected/b1485533-fd44-4182-a51a-8ed758b41442-kube-api-access-kqcsd\") pod \"b1485533-fd44-4182-a51a-8ed758b41442\" (UID: \"b1485533-fd44-4182-a51a-8ed758b41442\") " Oct 30 21:41:32.067870 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.067842 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1485533-fd44-4182-a51a-8ed758b41442-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1485533-fd44-4182-a51a-8ed758b41442-maas-api-openshift-ca\") pod \"b1485533-fd44-4182-a51a-8ed758b41442\" (UID: \"b1485533-fd44-4182-a51a-8ed758b41442\") " Oct 30 21:41:32.068119 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.068103 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1485533-fd44-4182-a51a-8ed758b41442-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1485533-fd44-4182-a51a-8ed758b41442-maas-api-xks-ca\") pod \"b1485533-fd44-4182-a51a-8ed758b41442\" (UID: \"b1485533-fd44-4182-a51a-8ed758b41442\") " Oct 30 21:41:32.069529 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.069506 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1485533-fd44-4182-a51a-8ed758b41442-maas-api-openshift-ca" pod "b1485533-fd44-4182-a51a-8ed758b41442" (UID: "b1485533-fd44-4182-a51a-8ed758b41442"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:32.074149 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.074129 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1485533-fd44-4182-a51a-8ed758b41442-maas-api-xks-ca" pod "b1485533-fd44-4182-a51a-8ed758b41442" (UID: "b1485533-fd44-4182-a51a-8ed758b41442"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:32.075963 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.075942 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:41:32.076068 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.075983 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:41:32.076068 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.076017 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:41:32.076068 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.076053 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:41:32.076234 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.076087 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:41:32.076234 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.076119 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:41:32.076234 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.076149 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:41:32.076234 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.076181 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:41:32.076234 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.076214 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:41:32.076481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.076244 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:41:32.076481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.076276 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:41:32.076481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.076309 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:41:32.076481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.076338 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:41:32.085101 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.085067 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:32.085215 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.085148 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:32.085215 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.085189 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:32.086926 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.086887 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1485533-fd44-4182-a51a-8ed758b41442-kube-api-access-kqcsd" pod "b1485533-fd44-4182-a51a-8ed758b41442" (UID: "b1485533-fd44-4182-a51a-8ed758b41442"). InnerVolumeSpecName "kube-api-access-kqcsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:32.138725 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.138698 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42"] Oct 30 21:41:32.146458 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.146436 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z"] Oct 30 21:41:32.155914 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.155866 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k"] Oct 30 21:41:32.163895 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.163868 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c701fda2-f82w4"] Oct 30 21:41:32.164625 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.164605 5801 cpu_manager.go:414] "containerMap: removing container" podUID="b1485533-fd44-4182-a51a-8ed758b41442" containerName="revoke-keys" Oct 30 21:41:32.164807 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.164742 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1485533-fd44-4182-a51a-8ed758b41442" containerName="revoke-keys" Oct 30 21:41:32.165055 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.165038 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1485533-fd44-4182-a51a-8ed758b41442" containerName="revoke-keys" Oct 30 21:41:32.173234 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.172849 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqcsd\" (UniqueName: \"kubernetes.io/projected/b1485533-fd44-4182-a51a-8ed758b41442-kube-api-access-kqcsd\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:32.173234 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.172879 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1485533-fd44-4182-a51a-8ed758b41442-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:32.173234 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.172897 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1485533-fd44-4182-a51a-8ed758b41442-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:32.173536 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.173503 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c701fda2-f82w4" Oct 30 21:41:32.178908 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.178789 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c701fda2-f82w4"] Oct 30 21:41:32.202888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.202840 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:32.232131 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.232091 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:32.232391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.232361 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:32.234138 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.216057 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4"] Oct 30 21:41:32.277679 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.276745 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5a6ffe-c3ea-4868-9ac5-6ed43040349e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c701fda2-f82w4\" (UID: \"fd5a6ffe-c3ea-4868-9ac5-6ed43040349e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c701fda2-f82w4" Oct 30 21:41:32.277679 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.276976 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmbgk\" (UniqueName: \"kubernetes.io/projected/fd5a6ffe-c3ea-4868-9ac5-6ed43040349e-kube-api-access-cmbgk\") pod \"maas-api-revoke-sub-c701fda2-f82w4\" (UID: \"fd5a6ffe-c3ea-4868-9ac5-6ed43040349e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c701fda2-f82w4" Oct 30 21:41:32.277679 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.277076 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5a6ffe-c3ea-4868-9ac5-6ed43040349e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c701fda2-f82w4\" (UID: \"fd5a6ffe-c3ea-4868-9ac5-6ed43040349e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c701fda2-f82w4" Oct 30 21:41:32.395075 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.393158 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmbgk\" (UniqueName: \"kubernetes.io/projected/fd5a6ffe-c3ea-4868-9ac5-6ed43040349e-kube-api-access-cmbgk\") pod \"maas-api-revoke-sub-c701fda2-f82w4\" (UID: \"fd5a6ffe-c3ea-4868-9ac5-6ed43040349e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c701fda2-f82w4" Oct 30 21:41:32.395075 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.393306 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5a6ffe-c3ea-4868-9ac5-6ed43040349e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c701fda2-f82w4\" (UID: \"fd5a6ffe-c3ea-4868-9ac5-6ed43040349e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c701fda2-f82w4" Oct 30 21:41:32.395075 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.393380 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5a6ffe-c3ea-4868-9ac5-6ed43040349e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c701fda2-f82w4\" (UID: \"fd5a6ffe-c3ea-4868-9ac5-6ed43040349e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c701fda2-f82w4" Oct 30 21:41:32.395075 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.394260 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5a6ffe-c3ea-4868-9ac5-6ed43040349e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c701fda2-f82w4\" (UID: \"fd5a6ffe-c3ea-4868-9ac5-6ed43040349e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c701fda2-f82w4" Oct 30 21:41:32.400671 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.400642 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5a6ffe-c3ea-4868-9ac5-6ed43040349e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c701fda2-f82w4\" (UID: \"fd5a6ffe-c3ea-4868-9ac5-6ed43040349e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c701fda2-f82w4" Oct 30 21:41:32.406574 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.406548 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmbgk\" (UniqueName: \"kubernetes.io/projected/fd5a6ffe-c3ea-4868-9ac5-6ed43040349e-kube-api-access-cmbgk\") pod \"maas-api-revoke-sub-c701fda2-f82w4\" (UID: \"fd5a6ffe-c3ea-4868-9ac5-6ed43040349e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c701fda2-f82w4" Oct 30 21:41:32.534222 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.524397 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:32.547183 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.546618 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c701fda2-f82w4" Oct 30 21:41:32.898292 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.898259 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c701fda2-f82w4"] Oct 30 21:41:32.992105 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.991367 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c701fda2-f82w4" event={"ID":"fd5a6ffe-c3ea-4868-9ac5-6ed43040349e","Type":"ContainerStarted","Data":"9afcc0ddd40dcbf8bf8af90589c27e7287679a44439a4399da18893c2c11c932"} Oct 30 21:41:32.994021 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.993995 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" event={"ID":"528a2150-a9c8-42d4-8203-113bc618287a","Type":"ContainerStarted","Data":"3eec4a313b1e11242469d9a71c03c5223525b005268d8da576e618346936b9c7"} Oct 30 21:41:32.994177 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.994048 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" event={"ID":"528a2150-a9c8-42d4-8203-113bc618287a","Type":"ContainerStarted","Data":"bf230e2376045525159c8b7653515ab77629c56d2c572077c54491af9ec2b087"} Oct 30 21:41:32.996224 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.996199 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" event={"ID":"6aaf74ef-1550-443a-a046-d08fc45e07f1","Type":"ContainerStarted","Data":"69b10569b4e2f336e6c7eea1aa64f4d05674b0081cb3afd71472129b19405761"} Oct 30 21:41:32.996325 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.996246 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" event={"ID":"6aaf74ef-1550-443a-a046-d08fc45e07f1","Type":"ContainerStarted","Data":"8eaf73101a498d2568e3a90dc769ff86debb32a1296fbd41e3ca48c3a9e8889f"} Oct 30 21:41:32.998073 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.998049 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" event={"ID":"96f41b1e-ff7a-47b1-990a-677cdcd42a7f","Type":"ContainerStarted","Data":"6d48376b75ae2bf1826e0d48b34bc11740a011779f380450350401cc4d588e5c"} Oct 30 21:41:32.998158 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.998087 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" event={"ID":"96f41b1e-ff7a-47b1-990a-677cdcd42a7f","Type":"ContainerStarted","Data":"8c7aaacb0e7317fcbd4d7868c1017f966eb5e4ceeece23434d04a608d6657c21"} Oct 30 21:41:32.999946 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.999922 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" event={"ID":"b8cc8b5b-1443-40da-a41d-745fcba8e474","Type":"ContainerStarted","Data":"65f4a6712c38dd9c3273bed541a9589206a7619bc98a1b557cf337bdb1107e9f"} Oct 30 21:41:33.000023 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:32.999967 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1caf5a9b-ndfhz" Oct 30 21:41:33.018849 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:33.018808 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" podStartSLOduration=10.018797286 podStartE2EDuration="10.018797286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:33.017463376 +0000 UTC m=+761.613280037" watchObservedRunningTime="2026-10-30 21:41:33.018797286 +0000 UTC m=+761.614613946" Oct 30 21:41:33.050054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:33.050001 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" podStartSLOduration=10.049984557 podStartE2EDuration="10.049984557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:33.04190614 +0000 UTC m=+761.637722800" watchObservedRunningTime="2026-10-30 21:41:33.049984557 +0000 UTC m=+761.645801220" Oct 30 21:41:33.065636 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:33.065587 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" podStartSLOduration=9.065576157 podStartE2EDuration="9.065576157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:33.063829923 +0000 UTC m=+761.659646623" watchObservedRunningTime="2026-10-30 21:41:33.065576157 +0000 UTC m=+761.661392816" Oct 30 21:41:33.103929 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:33.103876 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" podStartSLOduration=9.103861001 podStartE2EDuration="9.103861001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:33.099910499 +0000 UTC m=+761.695727162" watchObservedRunningTime="2026-10-30 21:41:33.103861001 +0000 UTC m=+761.699677661" Oct 30 21:41:33.167306 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:33.167272 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:33.168576 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:33.168552 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:41:33.168792 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:33.168595 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" podUID="b8cc8b5b-1443-40da-a41d-745fcba8e474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 21:41:33.425834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:33.424858 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:33.425834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:33.425002 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:41:33.425834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:33.425055 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" podUID="528a2150-a9c8-42d4-8203-113bc618287a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 21:41:33.562806 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:33.534006 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:33.581806 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:33.577923 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:41:33.581806 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:33.577986 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" podUID="96f41b1e-ff7a-47b1-990a-677cdcd42a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 21:41:34.022436 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.022396 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:41:34.026893 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.026846 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" podUID="6aaf74ef-1550-443a-a046-d08fc45e07f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 21:41:34.185302 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.185262 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:41:34.185599 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.185566 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" podUID="b8cc8b5b-1443-40da-a41d-745fcba8e474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 21:41:34.304846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.285387 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ac40742-qndwc"] Oct 30 21:41:34.382060 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.382024 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:34.382403 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.382387 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac40742-qndwc" Oct 30 21:41:34.416914 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.416874 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:34.447795 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.430883 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:41:34.447795 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.430959 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" podUID="528a2150-a9c8-42d4-8203-113bc618287a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 21:41:34.462795 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.460468 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c701fda2-f82w4" event={"ID":"fd5a6ffe-c3ea-4868-9ac5-6ed43040349e","Type":"ContainerStarted","Data":"beddd0e617d1c48d8c320829ef983d8c070f05b53c9abfcb47f692c191510c8f"} Oct 30 21:41:34.531778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.508343 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:34.543853 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.538961 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:41:34.543853 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.543142 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" podUID="96f41b1e-ff7a-47b1-990a-677cdcd42a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 21:41:34.633842 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.602495 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp7qs\" (UniqueName: \"kubernetes.io/projected/09502484-dcf7-4af4-a220-628ef74cb996-kube-api-access-fp7qs\") pod \"maas-api-revoke-sub-1ac40742-qndwc\" (UID: \"09502484-dcf7-4af4-a220-628ef74cb996\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac40742-qndwc" Oct 30 21:41:34.633842 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.602681 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09502484-dcf7-4af4-a220-628ef74cb996-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ac40742-qndwc\" (UID: \"09502484-dcf7-4af4-a220-628ef74cb996\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac40742-qndwc" Oct 30 21:41:34.633842 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.603334 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09502484-dcf7-4af4-a220-628ef74cb996-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ac40742-qndwc\" (UID: \"09502484-dcf7-4af4-a220-628ef74cb996\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac40742-qndwc" Oct 30 21:41:34.713778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.712825 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ac40742-qndwc"] Oct 30 21:41:34.717775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.714845 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09502484-dcf7-4af4-a220-628ef74cb996-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ac40742-qndwc\" (UID: \"09502484-dcf7-4af4-a220-628ef74cb996\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac40742-qndwc" Oct 30 21:41:34.717775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.715273 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09502484-dcf7-4af4-a220-628ef74cb996-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ac40742-qndwc\" (UID: \"09502484-dcf7-4af4-a220-628ef74cb996\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac40742-qndwc" Oct 30 21:41:34.717775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.715385 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp7qs\" (UniqueName: \"kubernetes.io/projected/09502484-dcf7-4af4-a220-628ef74cb996-kube-api-access-fp7qs\") pod \"maas-api-revoke-sub-1ac40742-qndwc\" (UID: \"09502484-dcf7-4af4-a220-628ef74cb996\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac40742-qndwc" Oct 30 21:41:34.740777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.734846 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09502484-dcf7-4af4-a220-628ef74cb996-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ac40742-qndwc\" (UID: \"09502484-dcf7-4af4-a220-628ef74cb996\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac40742-qndwc" Oct 30 21:41:34.828777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.828132 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09502484-dcf7-4af4-a220-628ef74cb996-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ac40742-qndwc\" (UID: \"09502484-dcf7-4af4-a220-628ef74cb996\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac40742-qndwc" Oct 30 21:41:34.853564 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.853525 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp7qs\" (UniqueName: \"kubernetes.io/projected/09502484-dcf7-4af4-a220-628ef74cb996-kube-api-access-fp7qs\") pod \"maas-api-revoke-sub-1ac40742-qndwc\" (UID: \"09502484-dcf7-4af4-a220-628ef74cb996\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac40742-qndwc" Oct 30 21:41:34.875777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.873934 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c701fda2-f82w4" podStartSLOduration=3.873909475 podStartE2EDuration="3.873909475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:34.506621236 +0000 UTC m=+763.102437922" watchObservedRunningTime="2026-10-30 21:41:34.873909475 +0000 UTC m=+763.469726135" Oct 30 21:41:34.906833 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.855676 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:34.927783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.925938 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:41:34.927783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.927678 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:41:34.928018 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.928004 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:41:34.931774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.928857 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:41:34.931774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.928915 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" podUID="6aaf74ef-1550-443a-a046-d08fc45e07f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 21:41:34.957023 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:34.956896 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:35.037778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.031850 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:35.104788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.083802 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:35.104788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.084138 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:41:35.104788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.084365 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:41:35.104788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.084591 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:35.104788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.098384 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:41:35.222855 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.190460 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac40742-qndwc" Oct 30 21:41:35.222855 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.190818 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:41:35.222855 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.190867 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" podUID="b8cc8b5b-1443-40da-a41d-745fcba8e474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 21:41:35.222855 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.215387 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:35.257361 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.254875 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:35.399935 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.399888 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:41:35.404789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.401809 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:35.455114 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.439267 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:41:35.455114 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.439646 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:41:35.510860 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.460079 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:41:35.510860 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.460237 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" podUID="528a2150-a9c8-42d4-8203-113bc618287a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 21:41:35.510860 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.505468 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:41:35.510860 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.505523 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" podUID="96f41b1e-ff7a-47b1-990a-677cdcd42a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 21:41:35.543829 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.543434 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:41:35.566526 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.566469 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:35.795880 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.795793 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:41:35.871103 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.871061 5801 generic.go:356] "Generic (PLEG): container finished" podID="fd5a6ffe-c3ea-4868-9ac5-6ed43040349e" containerID="beddd0e617d1c48d8c320829ef983d8c070f05b53c9abfcb47f692c191510c8f" exitCode=0 Oct 30 21:41:35.906815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.875925 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c701fda2-f82w4" event={"ID":"fd5a6ffe-c3ea-4868-9ac5-6ed43040349e","Type":"ContainerDied","Data":"beddd0e617d1c48d8c320829ef983d8c070f05b53c9abfcb47f692c191510c8f"} Oct 30 21:41:35.952077 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.932440 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:35.952395 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.932490 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:35.952912 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.932857 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:35.953482 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.953454 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:41:35.953646 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.953634 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:41:35.961105 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.933450 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:41:35.966892 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.966838 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" podUID="6aaf74ef-1550-443a-a046-d08fc45e07f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 21:41:36.014780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:35.983890 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:41:36.052219 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.052099 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:36.053238 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.053218 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:41:36.147509 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.136852 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:41:36.147509 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.137259 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:36.184051 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.162835 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:36.184051 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.163273 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:36.184051 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.163600 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:41:36.184051 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.163661 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:41:36.184051 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.163700 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:41:36.184051 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.163736 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:41:36.184051 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.183312 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded" Oct 30 21:41:36.327508 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.327471 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:41:36.330860 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.330817 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" podUID="b8cc8b5b-1443-40da-a41d-745fcba8e474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 21:41:36.404641 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.400407 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:36.404972 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.404944 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:41:36.433780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.407111 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:36.488773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.471854 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:41:36.515175 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.436024 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:41:36.515175 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.489870 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" podUID="528a2150-a9c8-42d4-8203-113bc618287a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 21:41:36.515175 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.465912 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:41:36.657642 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.600062 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:41:36.657642 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.600126 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" podUID="96f41b1e-ff7a-47b1-990a-677cdcd42a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 21:41:36.657642 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.600788 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:36.773789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.749296 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:37.011836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.968680 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:37.011836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.969063 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:37.011836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:36.969314 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:37.037821 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:37.029499 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:41:37.089827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:37.067844 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:37.116595 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:37.105329 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:41:37.116595 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:37.105387 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" podUID="6aaf74ef-1550-443a-a046-d08fc45e07f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 21:41:37.166016 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:37.150160 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:37.166016 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:37.151040 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:41:37.178271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:37.178227 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:37.195831 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:37.190902 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:37.334783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:37.333908 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:41:37.334783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:37.333964 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" podUID="b8cc8b5b-1443-40da-a41d-745fcba8e474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 21:41:37.424804 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:37.411020 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:37.740857 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:37.551915 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:37.740857 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:37.605194 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:41:37.740857 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:37.605255 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" podUID="528a2150-a9c8-42d4-8203-113bc618287a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 21:41:37.740857 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:37.608866 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:41:37.740857 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:37.608912 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" podUID="96f41b1e-ff7a-47b1-990a-677cdcd42a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 21:41:37.740857 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:37.660079 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:37.740857 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:37.689886 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:41:38.068316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:37.923086 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:41:38.068316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:37.923136 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" podUID="6aaf74ef-1550-443a-a046-d08fc45e07f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 21:41:38.068316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:37.964431 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:41:38.068316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:38.001227 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:38.068316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:38.001517 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:41:38.068316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:38.001690 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:41:38.341892 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:38.183151 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:38.341892 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:38.183558 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 21:41:38.341892 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:38.256291 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:41:38.341892 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:38.256347 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" podUID="b8cc8b5b-1443-40da-a41d-745fcba8e474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 21:41:38.341892 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:38.260244 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:41:38.341892 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:38.283165 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:41:38.341892 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:38.332852 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:38.403389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:38.384202 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:38.539801 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:38.520081 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:41:38.539801 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:38.520159 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" podUID="96f41b1e-ff7a-47b1-990a-677cdcd42a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 21:41:38.638079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:38.637955 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:41:38.638379 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:38.638329 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" podUID="528a2150-a9c8-42d4-8203-113bc618287a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 21:41:38.665791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:38.665721 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sk2kw" podUID="faea70fa-2fa1-45b8-9409-72fb5d589cc3" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.25:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:38.759782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:38.692739 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:41:38.798576 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:38.798525 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded" Oct 30 21:41:38.799084 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:38.799012 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:41:38.888862 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:38.816701 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:38.888862 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:38.831842 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:39.076092 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.076054 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:41:39.076422 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.076389 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" podUID="6aaf74ef-1550-443a-a046-d08fc45e07f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 21:41:39.267329 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.267284 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:39.269516 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.269471 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:41:39.274195 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.274150 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:41:39.284249 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.284213 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Oct 30 21:41:39.289143 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.289112 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:41:39.290361 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.290330 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 21:41:39.291502 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.291479 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:41:39.331448 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.331399 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:41:39.331699 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.331671 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" podUID="b8cc8b5b-1443-40da-a41d-745fcba8e474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 21:41:39.369811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.345883 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:41:39.419978 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.403518 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:39.419978 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.405888 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:39.459797 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.458584 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ac40742-qndwc"] Oct 30 21:41:39.504788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.503315 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:41:39.554999 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.551074 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:39.555632 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.555601 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:41:39.555939 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.555898 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" podUID="528a2150-a9c8-42d4-8203-113bc618287a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 21:41:39.781850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.658640 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:41:39.794914 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.794860 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" podUID="96f41b1e-ff7a-47b1-990a-677cdcd42a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 21:41:39.837289 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.837246 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:41:39.924262 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.839832 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:39.925271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.900601 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:39.925938 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.922694 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:41:39.926130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:39.926106 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" podUID="6aaf74ef-1550-443a-a046-d08fc45e07f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 21:41:40.222009 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.221976 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c701fda2-f82w4" Oct 30 21:41:40.290775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.290716 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:40.295374 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.295342 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:41:40.333201 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.333155 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:40.350049 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.347250 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:41:40.350728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.347294 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 21:41:40.351906 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.333094 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:40.383812 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:41:40.352081 5801 helpers.go:246] readString: Failed to read "/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09502484_dcf7_4af4_a220_628ef74cb996.slice/crio-ba4b8c68dcc027aba5e433788cfc5703e540a1d4e01aff511fb48d146a433745.scope/cpuset.cpus.effective": read /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09502484_dcf7_4af4_a220_628ef74cb996.slice/crio-ba4b8c68dcc027aba5e433788cfc5703e540a1d4e01aff511fb48d146a433745.scope/cpuset.cpus.effective: no such device Oct 30 21:41:40.436778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.405223 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:40.491444 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.481806 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:40.491444 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.482775 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:41:40.491444 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.482819 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" podUID="528a2150-a9c8-42d4-8203-113bc618287a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 21:41:40.513787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.506210 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:40.546246 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.546210 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:41:40.700898 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.576640 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd5a6ffe-c3ea-4868-9ac5-6ed43040349e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5a6ffe-c3ea-4868-9ac5-6ed43040349e-maas-api-openshift-ca\") pod \"fd5a6ffe-c3ea-4868-9ac5-6ed43040349e\" (UID: \"fd5a6ffe-c3ea-4868-9ac5-6ed43040349e\") " Oct 30 21:41:40.700898 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.676294 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd5a6ffe-c3ea-4868-9ac5-6ed43040349e-kube-api-access-cmbgk\" (UniqueName: \"kubernetes.io/projected/fd5a6ffe-c3ea-4868-9ac5-6ed43040349e-kube-api-access-cmbgk\") pod \"fd5a6ffe-c3ea-4868-9ac5-6ed43040349e\" (UID: \"fd5a6ffe-c3ea-4868-9ac5-6ed43040349e\") " Oct 30 21:41:40.700898 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.676380 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd5a6ffe-c3ea-4868-9ac5-6ed43040349e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5a6ffe-c3ea-4868-9ac5-6ed43040349e-maas-api-xks-ca\") pod \"fd5a6ffe-c3ea-4868-9ac5-6ed43040349e\" (UID: \"fd5a6ffe-c3ea-4868-9ac5-6ed43040349e\") " Oct 30 21:41:40.700898 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.577287 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5a6ffe-c3ea-4868-9ac5-6ed43040349e-maas-api-openshift-ca" pod "fd5a6ffe-c3ea-4868-9ac5-6ed43040349e" (UID: "fd5a6ffe-c3ea-4868-9ac5-6ed43040349e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:40.700898 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.669386 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:40.700898 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:41:40.593231 5801 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.064s" Oct 30 21:41:40.700898 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.689105 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:41:40.700898 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.689140 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac40742-qndwc" event={"ID":"09502484-dcf7-4af4-a220-628ef74cb996","Type":"ContainerStarted","Data":"6b71a6208f76fd819f8f2411619d686f782ee6dc294e01a8b327d14f39308066"} Oct 30 21:41:40.700898 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.689271 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:40.700898 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.689325 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:41:40.700898 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.693792 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:41:40.700898 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.669450 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" podUID="96f41b1e-ff7a-47b1-990a-677cdcd42a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 21:41:40.727860 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.727815 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:41:40.728119 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.728083 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" podUID="b8cc8b5b-1443-40da-a41d-745fcba8e474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 21:41:40.746231 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.746156 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:41:40.751351 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.750156 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:41:40.784878 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.783265 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5a6ffe-c3ea-4868-9ac5-6ed43040349e-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:40.795171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.794736 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:41:40.857812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.855177 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd5a6ffe-c3ea-4868-9ac5-6ed43040349e-maas-api-xks-ca" pod "fd5a6ffe-c3ea-4868-9ac5-6ed43040349e" (UID: "fd5a6ffe-c3ea-4868-9ac5-6ed43040349e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:40.860826 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.860743 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:41:40.883774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.878207 5801 generic.go:356] "Generic (PLEG): container finished" podID="09502484-dcf7-4af4-a220-628ef74cb996" containerID="ba4b8c68dcc027aba5e433788cfc5703e540a1d4e01aff511fb48d146a433745" exitCode=0 Oct 30 21:41:40.883774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.878366 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac40742-qndwc" event={"ID":"09502484-dcf7-4af4-a220-628ef74cb996","Type":"ContainerDied","Data":"ba4b8c68dcc027aba5e433788cfc5703e540a1d4e01aff511fb48d146a433745"} Oct 30 21:41:40.883774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.880719 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c701fda2-f82w4" Oct 30 21:41:40.883774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.881805 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c701fda2-f82w4" event={"ID":"fd5a6ffe-c3ea-4868-9ac5-6ed43040349e","Type":"ContainerDied","Data":"9afcc0ddd40dcbf8bf8af90589c27e7287679a44439a4399da18893c2c11c932"} Oct 30 21:41:40.883774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.881845 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9afcc0ddd40dcbf8bf8af90589c27e7287679a44439a4399da18893c2c11c932" Oct 30 21:41:40.888770 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.886775 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5a6ffe-c3ea-4868-9ac5-6ed43040349e-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:40.907622 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.906368 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:41:40.907958 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.907933 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5a6ffe-c3ea-4868-9ac5-6ed43040349e-kube-api-access-cmbgk" pod "fd5a6ffe-c3ea-4868-9ac5-6ed43040349e" (UID: "fd5a6ffe-c3ea-4868-9ac5-6ed43040349e"). InnerVolumeSpecName "kube-api-access-cmbgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:40.969345 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.969301 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:40.985094 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.985064 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:41:40.988057 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.988024 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:41:40.988190 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.988076 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" podUID="6aaf74ef-1550-443a-a046-d08fc45e07f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 21:41:40.988348 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.988307 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmbgk\" (UniqueName: \"kubernetes.io/projected/fd5a6ffe-c3ea-4868-9ac5-6ed43040349e-kube-api-access-cmbgk\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:40.989151 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:40.989098 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:41:41.013905 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.013817 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg"] Oct 30 21:41:41.019712 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.019685 5801 cpu_manager.go:414] "containerMap: removing container" podUID="fd5a6ffe-c3ea-4868-9ac5-6ed43040349e" containerName="revoke-keys" Oct 30 21:41:41.019946 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.019926 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd5a6ffe-c3ea-4868-9ac5-6ed43040349e" containerName="revoke-keys" Oct 30 21:41:41.039785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.032103 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd5a6ffe-c3ea-4868-9ac5-6ed43040349e" containerName="revoke-keys" Oct 30 21:41:41.106659 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.105032 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:41:41.106659 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.105089 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:41:41.106659 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.105123 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:41:41.106659 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.105139 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.106659 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.105144 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg"] Oct 30 21:41:41.106659 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.105259 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:41:41.169508 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.169456 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:41:41.169675 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.169540 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" podUID="b8cc8b5b-1443-40da-a41d-745fcba8e474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 21:41:41.197491 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.197453 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-credential-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.197798 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.197772 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311939a8-e35d-4636-b052-a368f0a41a9a-istiod-ca-cert\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.208646 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.207621 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-workload-certs\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.209992 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.209953 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311939a8-e35d-4636-b052-a368f0a41a9a-istio-token\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.210361 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.210317 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-istio-envoy\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.210606 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.210588 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-istio-data\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.211941 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.211829 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-workload-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.212207 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.212172 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311939a8-e35d-4636-b052-a368f0a41a9a-istio-podinfo\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.213689 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.213668 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpjwd\" (UniqueName: \"kubernetes.io/projected/311939a8-e35d-4636-b052-a368f0a41a9a-kube-api-access-mpjwd\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.317836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.317325 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-istio-envoy\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.317836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.317379 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-istio-data\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.325472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.322786 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-workload-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.325472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.322860 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311939a8-e35d-4636-b052-a368f0a41a9a-istio-podinfo\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.325472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.323009 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpjwd\" (UniqueName: \"kubernetes.io/projected/311939a8-e35d-4636-b052-a368f0a41a9a-kube-api-access-mpjwd\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.325472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.323055 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-credential-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.325472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.323103 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311939a8-e35d-4636-b052-a368f0a41a9a-istiod-ca-cert\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.325472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.323177 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-workload-certs\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.325472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.323202 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311939a8-e35d-4636-b052-a368f0a41a9a-istio-token\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.372954 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.372682 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-istio-data\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.373509 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.373471 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-workload-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.396674 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.396645 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311939a8-e35d-4636-b052-a368f0a41a9a-istio-podinfo\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.398029 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.398005 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311939a8-e35d-4636-b052-a368f0a41a9a-istiod-ca-cert\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.398421 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.398398 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-credential-socket\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.398846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.398826 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-workload-certs\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.417990 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.417958 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:41.499792 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.461457 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpjwd\" (UniqueName: \"kubernetes.io/projected/311939a8-e35d-4636-b052-a368f0a41a9a-kube-api-access-mpjwd\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.597825 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.571912 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-istio-envoy\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.665478 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.665441 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311939a8-e35d-4636-b052-a368f0a41a9a-istio-token\") pod \"e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:41.771776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:41.769025 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:42.004329 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.004290 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:41:42.004587 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.004556 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" podUID="6aaf74ef-1550-443a-a046-d08fc45e07f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 21:41:42.078927 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.077129 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d59c412-mhbzr"] Oct 30 21:41:42.138809 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.138777 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:42.139212 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.139194 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:42.141271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.141252 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d59c412-mhbzr" Oct 30 21:41:42.144189 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.144171 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:42.146157 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.146137 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:41:42.146862 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.146321 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:41:42.164379 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.164343 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d59c412-mhbzr"] Oct 30 21:41:42.245169 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.245136 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a079e35b-efc4-4cda-9cff-fcb6270ad56a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d59c412-mhbzr\" (UID: \"a079e35b-efc4-4cda-9cff-fcb6270ad56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d59c412-mhbzr" Oct 30 21:41:42.245346 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.245219 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2s6c\" (UniqueName: \"kubernetes.io/projected/a079e35b-efc4-4cda-9cff-fcb6270ad56a-kube-api-access-r2s6c\") pod \"maas-api-revoke-sub-1d59c412-mhbzr\" (UID: \"a079e35b-efc4-4cda-9cff-fcb6270ad56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d59c412-mhbzr" Oct 30 21:41:42.245346 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.245307 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a079e35b-efc4-4cda-9cff-fcb6270ad56a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d59c412-mhbzr\" (UID: \"a079e35b-efc4-4cda-9cff-fcb6270ad56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d59c412-mhbzr" Oct 30 21:41:42.291767 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.291719 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:42.349693 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.349639 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a079e35b-efc4-4cda-9cff-fcb6270ad56a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d59c412-mhbzr\" (UID: \"a079e35b-efc4-4cda-9cff-fcb6270ad56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d59c412-mhbzr" Oct 30 21:41:42.350289 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.350260 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a079e35b-efc4-4cda-9cff-fcb6270ad56a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d59c412-mhbzr\" (UID: \"a079e35b-efc4-4cda-9cff-fcb6270ad56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d59c412-mhbzr" Oct 30 21:41:42.350440 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.350421 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2s6c\" (UniqueName: \"kubernetes.io/projected/a079e35b-efc4-4cda-9cff-fcb6270ad56a-kube-api-access-r2s6c\") pod \"maas-api-revoke-sub-1d59c412-mhbzr\" (UID: \"a079e35b-efc4-4cda-9cff-fcb6270ad56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d59c412-mhbzr" Oct 30 21:41:42.351629 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.351605 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a079e35b-efc4-4cda-9cff-fcb6270ad56a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d59c412-mhbzr\" (UID: \"a079e35b-efc4-4cda-9cff-fcb6270ad56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d59c412-mhbzr" Oct 30 21:41:42.442292 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.442254 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:42.442712 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.442343 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:42.445156 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.444379 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:42.446314 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.446286 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2s6c\" (UniqueName: \"kubernetes.io/projected/a079e35b-efc4-4cda-9cff-fcb6270ad56a-kube-api-access-r2s6c\") pod \"maas-api-revoke-sub-1d59c412-mhbzr\" (UID: \"a079e35b-efc4-4cda-9cff-fcb6270ad56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d59c412-mhbzr" Oct 30 21:41:42.447397 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.447320 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a079e35b-efc4-4cda-9cff-fcb6270ad56a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d59c412-mhbzr\" (UID: \"a079e35b-efc4-4cda-9cff-fcb6270ad56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d59c412-mhbzr" Oct 30 21:41:42.502133 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.499358 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg"] Oct 30 21:41:42.593463 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.592636 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d59c412-mhbzr" Oct 30 21:41:42.712109 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.712084 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac40742-qndwc" Oct 30 21:41:42.859740 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.859703 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09502484-dcf7-4af4-a220-628ef74cb996-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09502484-dcf7-4af4-a220-628ef74cb996-maas-api-xks-ca\") pod \"09502484-dcf7-4af4-a220-628ef74cb996\" (UID: \"09502484-dcf7-4af4-a220-628ef74cb996\") " Oct 30 21:41:42.859916 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.859887 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09502484-dcf7-4af4-a220-628ef74cb996-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09502484-dcf7-4af4-a220-628ef74cb996-maas-api-openshift-ca\") pod \"09502484-dcf7-4af4-a220-628ef74cb996\" (UID: \"09502484-dcf7-4af4-a220-628ef74cb996\") " Oct 30 21:41:42.860076 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.860060 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09502484-dcf7-4af4-a220-628ef74cb996-kube-api-access-fp7qs\" (UniqueName: \"kubernetes.io/projected/09502484-dcf7-4af4-a220-628ef74cb996-kube-api-access-fp7qs\") pod \"09502484-dcf7-4af4-a220-628ef74cb996\" (UID: \"09502484-dcf7-4af4-a220-628ef74cb996\") " Oct 30 21:41:42.863799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.863731 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09502484-dcf7-4af4-a220-628ef74cb996-maas-api-openshift-ca" pod "09502484-dcf7-4af4-a220-628ef74cb996" (UID: "09502484-dcf7-4af4-a220-628ef74cb996"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:42.864956 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.864916 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09502484-dcf7-4af4-a220-628ef74cb996-maas-api-xks-ca" pod "09502484-dcf7-4af4-a220-628ef74cb996" (UID: "09502484-dcf7-4af4-a220-628ef74cb996"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:42.888701 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.888649 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09502484-dcf7-4af4-a220-628ef74cb996-kube-api-access-fp7qs" pod "09502484-dcf7-4af4-a220-628ef74cb996" (UID: "09502484-dcf7-4af4-a220-628ef74cb996"). InnerVolumeSpecName "kube-api-access-fp7qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:42.921566 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.921533 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:42.922336 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.922260 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:42.923886 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.923733 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:41:42.928188 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.925984 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:42.945285 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.945248 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:41:42.963291 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.963255 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp7qs\" (UniqueName: \"kubernetes.io/projected/09502484-dcf7-4af4-a220-628ef74cb996-kube-api-access-fp7qs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:42.963291 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.963292 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09502484-dcf7-4af4-a220-628ef74cb996-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:42.963583 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.963312 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09502484-dcf7-4af4-a220-628ef74cb996-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:42.981738 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.975883 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac40742-qndwc" Oct 30 21:41:42.981738 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.975909 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac40742-qndwc" event={"ID":"09502484-dcf7-4af4-a220-628ef74cb996","Type":"ContainerDied","Data":"6b71a6208f76fd819f8f2411619d686f782ee6dc294e01a8b327d14f39308066"} Oct 30 21:41:42.981738 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.975974 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b71a6208f76fd819f8f2411619d686f782ee6dc294e01a8b327d14f39308066" Oct 30 21:41:42.981738 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.978176 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" event={"ID":"311939a8-e35d-4636-b052-a368f0a41a9a","Type":"ContainerStarted","Data":"816b1d8caa5e9ecaf74350b88519d18975c33a8c70a7f4fd8bbfe3c2d182cbec"} Oct 30 21:41:42.981738 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:42.978227 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" event={"ID":"311939a8-e35d-4636-b052-a368f0a41a9a","Type":"ContainerStarted","Data":"0c9bc0c22bb9758354bff101d39a35aaa3c61c064673aa083536f00ea3edf0c1"} Oct 30 21:41:43.052030 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:43.051832 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" podStartSLOduration=4.051815595 podStartE2EDuration="4.051815595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:43.048580848 +0000 UTC m=+771.644397508" watchObservedRunningTime="2026-10-30 21:41:43.051815595 +0000 UTC m=+771.647632255" Oct 30 21:41:43.077655 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:43.077445 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d59c412-mhbzr"] Oct 30 21:41:43.263567 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:43.263520 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:43.774836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:43.773843 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 21:41:43.774836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:43.773909 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" podUID="311939a8-e35d-4636-b052-a368f0a41a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 21:41:43.774836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:43.774190 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:43.991176 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:43.991135 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d59c412-mhbzr" event={"ID":"a079e35b-efc4-4cda-9cff-fcb6270ad56a","Type":"ContainerStarted","Data":"906aa1b32975dd8fccf20c3e06703a1ac8b5c414bafb56941526b05cdeb7bb28"} Oct 30 21:41:43.991345 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:43.991203 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d59c412-mhbzr" event={"ID":"a079e35b-efc4-4cda-9cff-fcb6270ad56a","Type":"ContainerStarted","Data":"7cf6463684141576e67fa98124413c76577b72f923e1d78650b1cb7d995dac19"} Oct 30 21:41:44.014283 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:44.014233 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d59c412-mhbzr" podStartSLOduration=3.014220138 podStartE2EDuration="3.014220138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:44.010971756 +0000 UTC m=+772.606788436" watchObservedRunningTime="2026-10-30 21:41:44.014220138 +0000 UTC m=+772.610036797" Oct 30 21:41:44.120608 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:44.120575 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:44.693361 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:44.693322 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-105c3a96-dqjrf"] Oct 30 21:41:44.694024 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:44.694005 5801 cpu_manager.go:414] "containerMap: removing container" podUID="09502484-dcf7-4af4-a220-628ef74cb996" containerName="revoke-keys" Oct 30 21:41:44.694024 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:44.694023 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09502484-dcf7-4af4-a220-628ef74cb996" containerName="revoke-keys" Oct 30 21:41:44.694202 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:44.694132 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="09502484-dcf7-4af4-a220-628ef74cb996" containerName="revoke-keys" Oct 30 21:41:44.697678 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:44.697658 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105c3a96-dqjrf" Oct 30 21:41:44.704800 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:44.704729 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmlrb\" (UniqueName: \"kubernetes.io/projected/b156454b-be9d-43f3-9322-dc78aed291b9-kube-api-access-tmlrb\") pod \"maas-api-revoke-sub-105c3a96-dqjrf\" (UID: \"b156454b-be9d-43f3-9322-dc78aed291b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105c3a96-dqjrf" Oct 30 21:41:44.704936 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:44.704844 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b156454b-be9d-43f3-9322-dc78aed291b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-105c3a96-dqjrf\" (UID: \"b156454b-be9d-43f3-9322-dc78aed291b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105c3a96-dqjrf" Oct 30 21:41:44.704936 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:44.704916 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b156454b-be9d-43f3-9322-dc78aed291b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-105c3a96-dqjrf\" (UID: \"b156454b-be9d-43f3-9322-dc78aed291b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105c3a96-dqjrf" Oct 30 21:41:44.707481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:44.707083 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-105c3a96-dqjrf"] Oct 30 21:41:44.778194 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:44.778158 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 21:41:44.778570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:44.778214 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" podUID="311939a8-e35d-4636-b052-a368f0a41a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 21:41:44.806327 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:44.806294 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmlrb\" (UniqueName: \"kubernetes.io/projected/b156454b-be9d-43f3-9322-dc78aed291b9-kube-api-access-tmlrb\") pod \"maas-api-revoke-sub-105c3a96-dqjrf\" (UID: \"b156454b-be9d-43f3-9322-dc78aed291b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105c3a96-dqjrf" Oct 30 21:41:44.806481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:44.806359 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b156454b-be9d-43f3-9322-dc78aed291b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-105c3a96-dqjrf\" (UID: \"b156454b-be9d-43f3-9322-dc78aed291b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105c3a96-dqjrf" Oct 30 21:41:44.806481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:44.806399 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b156454b-be9d-43f3-9322-dc78aed291b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-105c3a96-dqjrf\" (UID: \"b156454b-be9d-43f3-9322-dc78aed291b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105c3a96-dqjrf" Oct 30 21:41:44.807015 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:44.806994 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b156454b-be9d-43f3-9322-dc78aed291b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-105c3a96-dqjrf\" (UID: \"b156454b-be9d-43f3-9322-dc78aed291b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105c3a96-dqjrf" Oct 30 21:41:44.808899 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:44.808872 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b156454b-be9d-43f3-9322-dc78aed291b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-105c3a96-dqjrf\" (UID: \"b156454b-be9d-43f3-9322-dc78aed291b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105c3a96-dqjrf" Oct 30 21:41:44.819834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:44.819812 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmlrb\" (UniqueName: \"kubernetes.io/projected/b156454b-be9d-43f3-9322-dc78aed291b9-kube-api-access-tmlrb\") pod \"maas-api-revoke-sub-105c3a96-dqjrf\" (UID: \"b156454b-be9d-43f3-9322-dc78aed291b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105c3a96-dqjrf" Oct 30 21:41:45.011219 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:45.011122 5801 generic.go:356] "Generic (PLEG): container finished" podID="a079e35b-efc4-4cda-9cff-fcb6270ad56a" containerID="906aa1b32975dd8fccf20c3e06703a1ac8b5c414bafb56941526b05cdeb7bb28" exitCode=0 Oct 30 21:41:45.011892 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:45.011867 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d59c412-mhbzr" event={"ID":"a079e35b-efc4-4cda-9cff-fcb6270ad56a","Type":"ContainerDied","Data":"906aa1b32975dd8fccf20c3e06703a1ac8b5c414bafb56941526b05cdeb7bb28"} Oct 30 21:41:45.023062 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:45.023037 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105c3a96-dqjrf" Oct 30 21:41:45.239415 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:45.239363 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:45.389890 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:45.389862 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-105c3a96-dqjrf"] Oct 30 21:41:45.778777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:45.778680 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 21:41:45.779265 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:45.778814 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" podUID="311939a8-e35d-4636-b052-a368f0a41a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 21:41:46.020999 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.020909 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105c3a96-dqjrf" event={"ID":"b156454b-be9d-43f3-9322-dc78aed291b9","Type":"ContainerStarted","Data":"0168f3c69ee2d307553a5b261953c138f6c338ba3008126bc110a7603ca9ffbb"} Oct 30 21:41:46.020999 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.020971 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105c3a96-dqjrf" event={"ID":"b156454b-be9d-43f3-9322-dc78aed291b9","Type":"ContainerStarted","Data":"e62e7f705bc8da9232581b9e3ebe313a8024af4d7689d07a98067ed0439ad105"} Oct 30 21:41:46.043654 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.043591 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105c3a96-dqjrf" podStartSLOduration=2.043577827 podStartE2EDuration="2.043577827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:46.039129023 +0000 UTC m=+774.634945709" watchObservedRunningTime="2026-10-30 21:41:46.043577827 +0000 UTC m=+774.639394487" Oct 30 21:41:46.222900 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.222874 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d59c412-mhbzr" Oct 30 21:41:46.229104 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.229082 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a079e35b-efc4-4cda-9cff-fcb6270ad56a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a079e35b-efc4-4cda-9cff-fcb6270ad56a-maas-api-openshift-ca\") pod \"a079e35b-efc4-4cda-9cff-fcb6270ad56a\" (UID: \"a079e35b-efc4-4cda-9cff-fcb6270ad56a\") " Oct 30 21:41:46.229213 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.229115 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a079e35b-efc4-4cda-9cff-fcb6270ad56a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a079e35b-efc4-4cda-9cff-fcb6270ad56a-maas-api-xks-ca\") pod \"a079e35b-efc4-4cda-9cff-fcb6270ad56a\" (UID: \"a079e35b-efc4-4cda-9cff-fcb6270ad56a\") " Oct 30 21:41:46.229213 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.229147 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a079e35b-efc4-4cda-9cff-fcb6270ad56a-kube-api-access-r2s6c\" (UniqueName: \"kubernetes.io/projected/a079e35b-efc4-4cda-9cff-fcb6270ad56a-kube-api-access-r2s6c\") pod \"a079e35b-efc4-4cda-9cff-fcb6270ad56a\" (UID: \"a079e35b-efc4-4cda-9cff-fcb6270ad56a\") " Oct 30 21:41:46.229425 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.229392 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a079e35b-efc4-4cda-9cff-fcb6270ad56a-maas-api-openshift-ca" pod "a079e35b-efc4-4cda-9cff-fcb6270ad56a" (UID: "a079e35b-efc4-4cda-9cff-fcb6270ad56a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:46.230057 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.230037 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a079e35b-efc4-4cda-9cff-fcb6270ad56a-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:46.231270 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.231250 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a079e35b-efc4-4cda-9cff-fcb6270ad56a-kube-api-access-r2s6c" pod "a079e35b-efc4-4cda-9cff-fcb6270ad56a" (UID: "a079e35b-efc4-4cda-9cff-fcb6270ad56a"). InnerVolumeSpecName "kube-api-access-r2s6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:46.231328 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.231309 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a079e35b-efc4-4cda-9cff-fcb6270ad56a-maas-api-xks-ca" pod "a079e35b-efc4-4cda-9cff-fcb6270ad56a" (UID: "a079e35b-efc4-4cda-9cff-fcb6270ad56a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:46.331601 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.331562 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a079e35b-efc4-4cda-9cff-fcb6270ad56a-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:46.331601 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.331598 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2s6c\" (UniqueName: \"kubernetes.io/projected/a079e35b-efc4-4cda-9cff-fcb6270ad56a-kube-api-access-r2s6c\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:46.340392 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.340362 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5706d53c-t2bst"] Oct 30 21:41:46.341004 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.340990 5801 cpu_manager.go:414] "containerMap: removing container" podUID="a079e35b-efc4-4cda-9cff-fcb6270ad56a" containerName="revoke-keys" Oct 30 21:41:46.341045 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.341006 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a079e35b-efc4-4cda-9cff-fcb6270ad56a" containerName="revoke-keys" Oct 30 21:41:46.341205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.341192 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="a079e35b-efc4-4cda-9cff-fcb6270ad56a" containerName="revoke-keys" Oct 30 21:41:46.345001 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.344980 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5706d53c-t2bst" Oct 30 21:41:46.382802 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.365425 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:46.389030 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.389001 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5706d53c-t2bst"] Oct 30 21:41:46.432497 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.432462 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nx8x\" (UniqueName: \"kubernetes.io/projected/701eda53-9940-4dd3-a0db-622c157347e7-kube-api-access-7nx8x\") pod \"maas-api-revoke-sub-5706d53c-t2bst\" (UID: \"701eda53-9940-4dd3-a0db-622c157347e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5706d53c-t2bst" Oct 30 21:41:46.432694 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.432641 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/701eda53-9940-4dd3-a0db-622c157347e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5706d53c-t2bst\" (UID: \"701eda53-9940-4dd3-a0db-622c157347e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5706d53c-t2bst" Oct 30 21:41:46.432797 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.432777 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/701eda53-9940-4dd3-a0db-622c157347e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5706d53c-t2bst\" (UID: \"701eda53-9940-4dd3-a0db-622c157347e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5706d53c-t2bst" Oct 30 21:41:46.534955 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.534909 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nx8x\" (UniqueName: \"kubernetes.io/projected/701eda53-9940-4dd3-a0db-622c157347e7-kube-api-access-7nx8x\") pod \"maas-api-revoke-sub-5706d53c-t2bst\" (UID: \"701eda53-9940-4dd3-a0db-622c157347e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5706d53c-t2bst" Oct 30 21:41:46.535909 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.535867 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/701eda53-9940-4dd3-a0db-622c157347e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5706d53c-t2bst\" (UID: \"701eda53-9940-4dd3-a0db-622c157347e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5706d53c-t2bst" Oct 30 21:41:46.536216 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.536177 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/701eda53-9940-4dd3-a0db-622c157347e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5706d53c-t2bst\" (UID: \"701eda53-9940-4dd3-a0db-622c157347e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5706d53c-t2bst" Oct 30 21:41:46.536952 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.536930 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/701eda53-9940-4dd3-a0db-622c157347e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5706d53c-t2bst\" (UID: \"701eda53-9940-4dd3-a0db-622c157347e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5706d53c-t2bst" Oct 30 21:41:46.538407 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.538382 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/701eda53-9940-4dd3-a0db-622c157347e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5706d53c-t2bst\" (UID: \"701eda53-9940-4dd3-a0db-622c157347e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5706d53c-t2bst" Oct 30 21:41:46.548191 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.548164 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nx8x\" (UniqueName: \"kubernetes.io/projected/701eda53-9940-4dd3-a0db-622c157347e7-kube-api-access-7nx8x\") pod \"maas-api-revoke-sub-5706d53c-t2bst\" (UID: \"701eda53-9940-4dd3-a0db-622c157347e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5706d53c-t2bst" Oct 30 21:41:46.694144 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.694064 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5706d53c-t2bst" Oct 30 21:41:46.780825 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.780308 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 21:41:46.780825 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:46.780381 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" podUID="311939a8-e35d-4636-b052-a368f0a41a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 21:41:47.001362 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:47.001333 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5706d53c-t2bst"] Oct 30 21:41:47.041823 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:47.041791 5801 generic.go:356] "Generic (PLEG): container finished" podID="b156454b-be9d-43f3-9322-dc78aed291b9" containerID="0168f3c69ee2d307553a5b261953c138f6c338ba3008126bc110a7603ca9ffbb" exitCode=0 Oct 30 21:41:47.042005 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:47.041980 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105c3a96-dqjrf" event={"ID":"b156454b-be9d-43f3-9322-dc78aed291b9","Type":"ContainerDied","Data":"0168f3c69ee2d307553a5b261953c138f6c338ba3008126bc110a7603ca9ffbb"} Oct 30 21:41:47.043486 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:47.043467 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d59c412-mhbzr" Oct 30 21:41:47.043574 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:47.043484 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d59c412-mhbzr" event={"ID":"a079e35b-efc4-4cda-9cff-fcb6270ad56a","Type":"ContainerDied","Data":"7cf6463684141576e67fa98124413c76577b72f923e1d78650b1cb7d995dac19"} Oct 30 21:41:47.043574 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:47.043526 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cf6463684141576e67fa98124413c76577b72f923e1d78650b1cb7d995dac19" Oct 30 21:41:47.045468 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:47.045447 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5706d53c-t2bst" event={"ID":"701eda53-9940-4dd3-a0db-622c157347e7","Type":"ContainerStarted","Data":"89c20a3b40b29e8122b77a351ad13c3a8e30e91def8a7656f50b1202c70a87a3"} Oct 30 21:41:47.526856 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:47.526821 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:47.772829 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:47.770142 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 21:41:47.772829 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:47.770206 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" podUID="311939a8-e35d-4636-b052-a368f0a41a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 21:41:48.082540 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:48.082510 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5706d53c-t2bst" event={"ID":"701eda53-9940-4dd3-a0db-622c157347e7","Type":"ContainerStarted","Data":"c33be47b56c0a8073fe4d666eb40d9daf24a92ec2a2f7bc7c5385843701d7e4d"} Oct 30 21:41:48.101056 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:48.100955 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5706d53c-t2bst" podStartSLOduration=2.100941035 podStartE2EDuration="2.100941035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:48.100246702 +0000 UTC m=+776.696063363" watchObservedRunningTime="2026-10-30 21:41:48.100941035 +0000 UTC m=+776.696757694" Oct 30 21:41:48.286468 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:48.286175 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105c3a96-dqjrf" Oct 30 21:41:48.385383 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:48.385284 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b156454b-be9d-43f3-9322-dc78aed291b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b156454b-be9d-43f3-9322-dc78aed291b9-maas-api-xks-ca\") pod \"b156454b-be9d-43f3-9322-dc78aed291b9\" (UID: \"b156454b-be9d-43f3-9322-dc78aed291b9\") " Oct 30 21:41:48.385569 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:48.385385 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b156454b-be9d-43f3-9322-dc78aed291b9-kube-api-access-tmlrb\" (UniqueName: \"kubernetes.io/projected/b156454b-be9d-43f3-9322-dc78aed291b9-kube-api-access-tmlrb\") pod \"b156454b-be9d-43f3-9322-dc78aed291b9\" (UID: \"b156454b-be9d-43f3-9322-dc78aed291b9\") " Oct 30 21:41:48.385635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:48.385564 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b156454b-be9d-43f3-9322-dc78aed291b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b156454b-be9d-43f3-9322-dc78aed291b9-maas-api-openshift-ca\") pod \"b156454b-be9d-43f3-9322-dc78aed291b9\" (UID: \"b156454b-be9d-43f3-9322-dc78aed291b9\") " Oct 30 21:41:48.386353 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:48.386315 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b156454b-be9d-43f3-9322-dc78aed291b9-maas-api-openshift-ca" pod "b156454b-be9d-43f3-9322-dc78aed291b9" (UID: "b156454b-be9d-43f3-9322-dc78aed291b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:48.388353 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:48.388326 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b156454b-be9d-43f3-9322-dc78aed291b9-maas-api-xks-ca" pod "b156454b-be9d-43f3-9322-dc78aed291b9" (UID: "b156454b-be9d-43f3-9322-dc78aed291b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:48.389593 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:48.389570 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b156454b-be9d-43f3-9322-dc78aed291b9-kube-api-access-tmlrb" pod "b156454b-be9d-43f3-9322-dc78aed291b9" (UID: "b156454b-be9d-43f3-9322-dc78aed291b9"). InnerVolumeSpecName "kube-api-access-tmlrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:48.434729 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:48.434529 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:48.509779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:48.509118 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b156454b-be9d-43f3-9322-dc78aed291b9-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:48.509779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:48.509156 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b156454b-be9d-43f3-9322-dc78aed291b9-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:48.509779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:48.509172 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmlrb\" (UniqueName: \"kubernetes.io/projected/b156454b-be9d-43f3-9322-dc78aed291b9-kube-api-access-tmlrb\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:48.778628 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:48.778118 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 21:41:48.778628 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:48.778179 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" podUID="311939a8-e35d-4636-b052-a368f0a41a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 21:41:49.093795 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:49.093742 5801 generic.go:356] "Generic (PLEG): container finished" podID="701eda53-9940-4dd3-a0db-622c157347e7" containerID="c33be47b56c0a8073fe4d666eb40d9daf24a92ec2a2f7bc7c5385843701d7e4d" exitCode=0 Oct 30 21:41:49.094205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:49.093823 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5706d53c-t2bst" event={"ID":"701eda53-9940-4dd3-a0db-622c157347e7","Type":"ContainerDied","Data":"c33be47b56c0a8073fe4d666eb40d9daf24a92ec2a2f7bc7c5385843701d7e4d"} Oct 30 21:41:49.095493 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:49.095474 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105c3a96-dqjrf" Oct 30 21:41:49.095643 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:49.095470 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105c3a96-dqjrf" event={"ID":"b156454b-be9d-43f3-9322-dc78aed291b9","Type":"ContainerDied","Data":"e62e7f705bc8da9232581b9e3ebe313a8024af4d7689d07a98067ed0439ad105"} Oct 30 21:41:49.095643 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:49.095615 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e62e7f705bc8da9232581b9e3ebe313a8024af4d7689d07a98067ed0439ad105" Oct 30 21:41:49.231386 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:49.231355 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:49.770707 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:49.770671 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 21:41:49.770910 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:49.770734 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" podUID="311939a8-e35d-4636-b052-a368f0a41a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 21:41:49.912670 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:49.912630 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d987ad-tt9nb"] Oct 30 21:41:49.913287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:49.913272 5801 cpu_manager.go:414] "containerMap: removing container" podUID="b156454b-be9d-43f3-9322-dc78aed291b9" containerName="revoke-keys" Oct 30 21:41:49.913333 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:49.913288 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b156454b-be9d-43f3-9322-dc78aed291b9" containerName="revoke-keys" Oct 30 21:41:49.913406 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:49.913396 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="b156454b-be9d-43f3-9322-dc78aed291b9" containerName="revoke-keys" Oct 30 21:41:49.917235 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:49.917219 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d987ad-tt9nb" Oct 30 21:41:49.927009 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:49.926983 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d987ad-tt9nb"] Oct 30 21:41:49.932159 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:49.932119 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5acfa7-3679-4ddf-91d4-db8d650f84b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01d987ad-tt9nb\" (UID: \"7d5acfa7-3679-4ddf-91d4-db8d650f84b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d987ad-tt9nb" Oct 30 21:41:49.932320 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:49.932253 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5acfa7-3679-4ddf-91d4-db8d650f84b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01d987ad-tt9nb\" (UID: \"7d5acfa7-3679-4ddf-91d4-db8d650f84b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d987ad-tt9nb" Oct 30 21:41:49.932435 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:49.932413 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srjcw\" (UniqueName: \"kubernetes.io/projected/7d5acfa7-3679-4ddf-91d4-db8d650f84b3-kube-api-access-srjcw\") pod \"maas-api-revoke-sub-01d987ad-tt9nb\" (UID: \"7d5acfa7-3679-4ddf-91d4-db8d650f84b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d987ad-tt9nb" Oct 30 21:41:50.034084 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:50.033982 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5acfa7-3679-4ddf-91d4-db8d650f84b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01d987ad-tt9nb\" (UID: \"7d5acfa7-3679-4ddf-91d4-db8d650f84b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d987ad-tt9nb" Oct 30 21:41:50.034084 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:50.034079 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5acfa7-3679-4ddf-91d4-db8d650f84b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01d987ad-tt9nb\" (UID: \"7d5acfa7-3679-4ddf-91d4-db8d650f84b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d987ad-tt9nb" Oct 30 21:41:50.034315 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:50.034206 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srjcw\" (UniqueName: \"kubernetes.io/projected/7d5acfa7-3679-4ddf-91d4-db8d650f84b3-kube-api-access-srjcw\") pod \"maas-api-revoke-sub-01d987ad-tt9nb\" (UID: \"7d5acfa7-3679-4ddf-91d4-db8d650f84b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d987ad-tt9nb" Oct 30 21:41:50.034932 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:50.034893 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5acfa7-3679-4ddf-91d4-db8d650f84b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01d987ad-tt9nb\" (UID: \"7d5acfa7-3679-4ddf-91d4-db8d650f84b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d987ad-tt9nb" Oct 30 21:41:50.036559 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:50.036538 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5acfa7-3679-4ddf-91d4-db8d650f84b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01d987ad-tt9nb\" (UID: \"7d5acfa7-3679-4ddf-91d4-db8d650f84b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d987ad-tt9nb" Oct 30 21:41:50.046580 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:50.046549 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srjcw\" (UniqueName: \"kubernetes.io/projected/7d5acfa7-3679-4ddf-91d4-db8d650f84b3-kube-api-access-srjcw\") pod \"maas-api-revoke-sub-01d987ad-tt9nb\" (UID: \"7d5acfa7-3679-4ddf-91d4-db8d650f84b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d987ad-tt9nb" Oct 30 21:41:50.158834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:50.158792 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:50.276774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:50.274709 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d987ad-tt9nb" Oct 30 21:41:50.376083 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:50.375739 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5706d53c-t2bst" Oct 30 21:41:50.567037 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:50.567006 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/701eda53-9940-4dd3-a0db-622c157347e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/701eda53-9940-4dd3-a0db-622c157347e7-maas-api-xks-ca\") pod \"701eda53-9940-4dd3-a0db-622c157347e7\" (UID: \"701eda53-9940-4dd3-a0db-622c157347e7\") " Oct 30 21:41:50.567225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:50.567201 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/701eda53-9940-4dd3-a0db-622c157347e7-kube-api-access-7nx8x\" (UniqueName: \"kubernetes.io/projected/701eda53-9940-4dd3-a0db-622c157347e7-kube-api-access-7nx8x\") pod \"701eda53-9940-4dd3-a0db-622c157347e7\" (UID: \"701eda53-9940-4dd3-a0db-622c157347e7\") " Oct 30 21:41:50.567348 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:50.567335 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/701eda53-9940-4dd3-a0db-622c157347e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/701eda53-9940-4dd3-a0db-622c157347e7-maas-api-openshift-ca\") pod \"701eda53-9940-4dd3-a0db-622c157347e7\" (UID: \"701eda53-9940-4dd3-a0db-622c157347e7\") " Oct 30 21:41:50.567672 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:50.567649 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/701eda53-9940-4dd3-a0db-622c157347e7-maas-api-openshift-ca" pod "701eda53-9940-4dd3-a0db-622c157347e7" (UID: "701eda53-9940-4dd3-a0db-622c157347e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:50.567977 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:50.567961 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/701eda53-9940-4dd3-a0db-622c157347e7-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:50.569300 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:50.569277 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/701eda53-9940-4dd3-a0db-622c157347e7-maas-api-xks-ca" pod "701eda53-9940-4dd3-a0db-622c157347e7" (UID: "701eda53-9940-4dd3-a0db-622c157347e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:50.569364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:50.569297 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701eda53-9940-4dd3-a0db-622c157347e7-kube-api-access-7nx8x" pod "701eda53-9940-4dd3-a0db-622c157347e7" (UID: "701eda53-9940-4dd3-a0db-622c157347e7"). InnerVolumeSpecName "kube-api-access-7nx8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:50.635891 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:41:50.635849 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5acfa7_3679_4ddf_91d4_db8d650f84b3.slice/crio-9570901d6541fc526e282430cf68836db55d1c0ba5f8e13c4b2f4ce44035f955 WatchSource:0}: Error finding container 9570901d6541fc526e282430cf68836db55d1c0ba5f8e13c4b2f4ce44035f955: Status 404 returned error can't find the container with id 9570901d6541fc526e282430cf68836db55d1c0ba5f8e13c4b2f4ce44035f955 Oct 30 21:41:50.637778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:50.637733 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d987ad-tt9nb"] Oct 30 21:41:50.669387 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:50.669355 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/701eda53-9940-4dd3-a0db-622c157347e7-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:50.669387 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:50.669383 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nx8x\" (UniqueName: \"kubernetes.io/projected/701eda53-9940-4dd3-a0db-622c157347e7-kube-api-access-7nx8x\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:50.774316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:50.774285 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:50.774857 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:50.774781 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:50.775531 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:50.775515 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:41:51.038599 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:51.038558 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:51.118036 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:51.117959 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5706d53c-t2bst" Oct 30 21:41:51.118036 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:51.117978 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5706d53c-t2bst" event={"ID":"701eda53-9940-4dd3-a0db-622c157347e7","Type":"ContainerDied","Data":"89c20a3b40b29e8122b77a351ad13c3a8e30e91def8a7656f50b1202c70a87a3"} Oct 30 21:41:51.118264 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:51.118067 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89c20a3b40b29e8122b77a351ad13c3a8e30e91def8a7656f50b1202c70a87a3" Oct 30 21:41:51.119997 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:51.119965 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d987ad-tt9nb" event={"ID":"7d5acfa7-3679-4ddf-91d4-db8d650f84b3","Type":"ContainerStarted","Data":"8b3384ffb9f510156233ceeddf5b2c50e5d47835fed2b6a5cdc9e4184c83f854"} Oct 30 21:41:51.120130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:51.120013 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d987ad-tt9nb" event={"ID":"7d5acfa7-3679-4ddf-91d4-db8d650f84b3","Type":"ContainerStarted","Data":"9570901d6541fc526e282430cf68836db55d1c0ba5f8e13c4b2f4ce44035f955"} Oct 30 21:41:51.185952 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:51.185767 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d987ad-tt9nb" podStartSLOduration=2.18573059 podStartE2EDuration="2.18573059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:51.151859647 +0000 UTC m=+779.747676334" watchObservedRunningTime="2026-10-30 21:41:51.18573059 +0000 UTC m=+779.781547251" Oct 30 21:41:51.749014 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:51.748971 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3003efe5-ppb5f"] Oct 30 21:41:51.749644 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:51.749631 5801 cpu_manager.go:414] "containerMap: removing container" podUID="701eda53-9940-4dd3-a0db-622c157347e7" containerName="revoke-keys" Oct 30 21:41:51.749688 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:51.749646 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="701eda53-9940-4dd3-a0db-622c157347e7" containerName="revoke-keys" Oct 30 21:41:51.749784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:51.749774 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="701eda53-9940-4dd3-a0db-622c157347e7" containerName="revoke-keys" Oct 30 21:41:51.753542 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:51.753526 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3003efe5-ppb5f" Oct 30 21:41:51.763545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:51.763240 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3003efe5-ppb5f"] Oct 30 21:41:51.791873 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:51.791832 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l9x4\" (UniqueName: \"kubernetes.io/projected/f264c6aa-54c1-41cd-9012-334aef4c85f2-kube-api-access-8l9x4\") pod \"maas-api-revoke-sub-3003efe5-ppb5f\" (UID: \"f264c6aa-54c1-41cd-9012-334aef4c85f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3003efe5-ppb5f" Oct 30 21:41:51.792031 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:51.791883 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f264c6aa-54c1-41cd-9012-334aef4c85f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3003efe5-ppb5f\" (UID: \"f264c6aa-54c1-41cd-9012-334aef4c85f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3003efe5-ppb5f" Oct 30 21:41:51.792117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:51.792072 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f264c6aa-54c1-41cd-9012-334aef4c85f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3003efe5-ppb5f\" (UID: \"f264c6aa-54c1-41cd-9012-334aef4c85f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3003efe5-ppb5f" Oct 30 21:41:51.800543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:51.800509 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:51.893388 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:51.893352 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l9x4\" (UniqueName: \"kubernetes.io/projected/f264c6aa-54c1-41cd-9012-334aef4c85f2-kube-api-access-8l9x4\") pod \"maas-api-revoke-sub-3003efe5-ppb5f\" (UID: \"f264c6aa-54c1-41cd-9012-334aef4c85f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3003efe5-ppb5f" Oct 30 21:41:51.893566 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:51.893418 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f264c6aa-54c1-41cd-9012-334aef4c85f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3003efe5-ppb5f\" (UID: \"f264c6aa-54c1-41cd-9012-334aef4c85f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3003efe5-ppb5f" Oct 30 21:41:51.893566 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:51.893530 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f264c6aa-54c1-41cd-9012-334aef4c85f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3003efe5-ppb5f\" (UID: \"f264c6aa-54c1-41cd-9012-334aef4c85f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3003efe5-ppb5f" Oct 30 21:41:51.894194 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:51.894169 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f264c6aa-54c1-41cd-9012-334aef4c85f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3003efe5-ppb5f\" (UID: \"f264c6aa-54c1-41cd-9012-334aef4c85f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3003efe5-ppb5f" Oct 30 21:41:51.896216 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:51.896196 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f264c6aa-54c1-41cd-9012-334aef4c85f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3003efe5-ppb5f\" (UID: \"f264c6aa-54c1-41cd-9012-334aef4c85f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3003efe5-ppb5f" Oct 30 21:41:51.906165 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:51.906105 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l9x4\" (UniqueName: \"kubernetes.io/projected/f264c6aa-54c1-41cd-9012-334aef4c85f2-kube-api-access-8l9x4\") pod \"maas-api-revoke-sub-3003efe5-ppb5f\" (UID: \"f264c6aa-54c1-41cd-9012-334aef4c85f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3003efe5-ppb5f" Oct 30 21:41:52.075180 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:52.075148 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3003efe5-ppb5f" Oct 30 21:41:52.130999 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:52.130964 5801 generic.go:356] "Generic (PLEG): container finished" podID="7d5acfa7-3679-4ddf-91d4-db8d650f84b3" containerID="8b3384ffb9f510156233ceeddf5b2c50e5d47835fed2b6a5cdc9e4184c83f854" exitCode=0 Oct 30 21:41:52.131184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:52.131051 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d987ad-tt9nb" event={"ID":"7d5acfa7-3679-4ddf-91d4-db8d650f84b3","Type":"ContainerDied","Data":"8b3384ffb9f510156233ceeddf5b2c50e5d47835fed2b6a5cdc9e4184c83f854"} Oct 30 21:41:52.366164 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:52.366129 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3003efe5-ppb5f"] Oct 30 21:41:52.384008 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:41:52.383957 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf264c6aa_54c1_41cd_9012_334aef4c85f2.slice/crio-23b87253af31471edd03f694ec0908d28a850f8e485e23513ecbb184e54933c5 WatchSource:0}: Error finding container 23b87253af31471edd03f694ec0908d28a850f8e485e23513ecbb184e54933c5: Status 404 returned error can't find the container with id 23b87253af31471edd03f694ec0908d28a850f8e485e23513ecbb184e54933c5 Oct 30 21:41:52.558628 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:52.558595 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed945f27-nrnqc"] Oct 30 21:41:52.563835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:52.563807 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed945f27-nrnqc" Oct 30 21:41:52.571711 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:52.570351 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed945f27-nrnqc"] Oct 30 21:41:52.609814 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:52.609703 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a7b59b3-0b71-4359-8918-b0ed7a8d1496-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed945f27-nrnqc\" (UID: \"8a7b59b3-0b71-4359-8918-b0ed7a8d1496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed945f27-nrnqc" Oct 30 21:41:52.609988 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:52.609897 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a7b59b3-0b71-4359-8918-b0ed7a8d1496-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed945f27-nrnqc\" (UID: \"8a7b59b3-0b71-4359-8918-b0ed7a8d1496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed945f27-nrnqc" Oct 30 21:41:52.609988 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:52.609948 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9lrp\" (UniqueName: \"kubernetes.io/projected/8a7b59b3-0b71-4359-8918-b0ed7a8d1496-kube-api-access-k9lrp\") pod \"maas-api-revoke-sub-ed945f27-nrnqc\" (UID: \"8a7b59b3-0b71-4359-8918-b0ed7a8d1496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed945f27-nrnqc" Oct 30 21:41:52.645808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:52.645746 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:52.712645 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:52.711704 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a7b59b3-0b71-4359-8918-b0ed7a8d1496-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed945f27-nrnqc\" (UID: \"8a7b59b3-0b71-4359-8918-b0ed7a8d1496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed945f27-nrnqc" Oct 30 21:41:52.712645 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:52.711787 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9lrp\" (UniqueName: \"kubernetes.io/projected/8a7b59b3-0b71-4359-8918-b0ed7a8d1496-kube-api-access-k9lrp\") pod \"maas-api-revoke-sub-ed945f27-nrnqc\" (UID: \"8a7b59b3-0b71-4359-8918-b0ed7a8d1496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed945f27-nrnqc" Oct 30 21:41:52.712645 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:52.712153 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a7b59b3-0b71-4359-8918-b0ed7a8d1496-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed945f27-nrnqc\" (UID: \"8a7b59b3-0b71-4359-8918-b0ed7a8d1496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed945f27-nrnqc" Oct 30 21:41:52.713173 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:52.713133 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a7b59b3-0b71-4359-8918-b0ed7a8d1496-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed945f27-nrnqc\" (UID: \"8a7b59b3-0b71-4359-8918-b0ed7a8d1496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed945f27-nrnqc" Oct 30 21:41:52.714819 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:52.714788 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a7b59b3-0b71-4359-8918-b0ed7a8d1496-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed945f27-nrnqc\" (UID: \"8a7b59b3-0b71-4359-8918-b0ed7a8d1496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed945f27-nrnqc" Oct 30 21:41:52.724863 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:52.724808 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9lrp\" (UniqueName: \"kubernetes.io/projected/8a7b59b3-0b71-4359-8918-b0ed7a8d1496-kube-api-access-k9lrp\") pod \"maas-api-revoke-sub-ed945f27-nrnqc\" (UID: \"8a7b59b3-0b71-4359-8918-b0ed7a8d1496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed945f27-nrnqc" Oct 30 21:41:52.896391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:52.896296 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed945f27-nrnqc" Oct 30 21:41:53.141528 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:53.141491 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3003efe5-ppb5f" event={"ID":"f264c6aa-54c1-41cd-9012-334aef4c85f2","Type":"ContainerStarted","Data":"99592a71e1a630006caeb084d4ae81189497a4cb047dc13253f7fe9d15be2199"} Oct 30 21:41:53.141697 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:53.141582 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3003efe5-ppb5f" event={"ID":"f264c6aa-54c1-41cd-9012-334aef4c85f2","Type":"ContainerStarted","Data":"23b87253af31471edd03f694ec0908d28a850f8e485e23513ecbb184e54933c5"} Oct 30 21:41:53.163067 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:53.162974 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3003efe5-ppb5f" podStartSLOduration=2.162958758 podStartE2EDuration="2.162958758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:53.161366151 +0000 UTC m=+781.757182857" watchObservedRunningTime="2026-10-30 21:41:53.162958758 +0000 UTC m=+781.758775418" Oct 30 21:41:53.182141 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:41:53.182086 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7b59b3_0b71_4359_8918_b0ed7a8d1496.slice/crio-2cc0ca06b1c89cb66fea6df9d89db8b81a891512be3ff70e03f228a88c990b7b WatchSource:0}: Error finding container 2cc0ca06b1c89cb66fea6df9d89db8b81a891512be3ff70e03f228a88c990b7b: Status 404 returned error can't find the container with id 2cc0ca06b1c89cb66fea6df9d89db8b81a891512be3ff70e03f228a88c990b7b Oct 30 21:41:53.184680 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:53.184654 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed945f27-nrnqc"] Oct 30 21:41:53.479046 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:53.478978 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d987ad-tt9nb" Oct 30 21:41:53.523447 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:53.523414 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d5acfa7-3679-4ddf-91d4-db8d650f84b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5acfa7-3679-4ddf-91d4-db8d650f84b3-maas-api-openshift-ca\") pod \"7d5acfa7-3679-4ddf-91d4-db8d650f84b3\" (UID: \"7d5acfa7-3679-4ddf-91d4-db8d650f84b3\") " Oct 30 21:41:53.523634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:53.523562 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d5acfa7-3679-4ddf-91d4-db8d650f84b3-kube-api-access-srjcw\" (UniqueName: \"kubernetes.io/projected/7d5acfa7-3679-4ddf-91d4-db8d650f84b3-kube-api-access-srjcw\") pod \"7d5acfa7-3679-4ddf-91d4-db8d650f84b3\" (UID: \"7d5acfa7-3679-4ddf-91d4-db8d650f84b3\") " Oct 30 21:41:53.523707 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:53.523633 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d5acfa7-3679-4ddf-91d4-db8d650f84b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5acfa7-3679-4ddf-91d4-db8d650f84b3-maas-api-xks-ca\") pod \"7d5acfa7-3679-4ddf-91d4-db8d650f84b3\" (UID: \"7d5acfa7-3679-4ddf-91d4-db8d650f84b3\") " Oct 30 21:41:53.523862 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:53.523838 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d5acfa7-3679-4ddf-91d4-db8d650f84b3-maas-api-openshift-ca" pod "7d5acfa7-3679-4ddf-91d4-db8d650f84b3" (UID: "7d5acfa7-3679-4ddf-91d4-db8d650f84b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:53.524489 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:53.524463 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5acfa7-3679-4ddf-91d4-db8d650f84b3-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:53.525948 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:53.525920 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5acfa7-3679-4ddf-91d4-db8d650f84b3-kube-api-access-srjcw" pod "7d5acfa7-3679-4ddf-91d4-db8d650f84b3" (UID: "7d5acfa7-3679-4ddf-91d4-db8d650f84b3"). InnerVolumeSpecName "kube-api-access-srjcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:53.526066 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:53.526024 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d5acfa7-3679-4ddf-91d4-db8d650f84b3-maas-api-xks-ca" pod "7d5acfa7-3679-4ddf-91d4-db8d650f84b3" (UID: "7d5acfa7-3679-4ddf-91d4-db8d650f84b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:53.551267 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:53.551235 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:53.626138 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:53.626102 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srjcw\" (UniqueName: \"kubernetes.io/projected/7d5acfa7-3679-4ddf-91d4-db8d650f84b3-kube-api-access-srjcw\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:53.626138 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:53.626135 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5acfa7-3679-4ddf-91d4-db8d650f84b3-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:54.034845 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.034810 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc"] Oct 30 21:41:54.035972 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.035945 5801 cpu_manager.go:414] "containerMap: removing container" podUID="7d5acfa7-3679-4ddf-91d4-db8d650f84b3" containerName="revoke-keys" Oct 30 21:41:54.035972 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.035970 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d5acfa7-3679-4ddf-91d4-db8d650f84b3" containerName="revoke-keys" Oct 30 21:41:54.036181 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.036163 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d5acfa7-3679-4ddf-91d4-db8d650f84b3" containerName="revoke-keys" Oct 30 21:41:54.041111 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.041088 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.055175 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.053077 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc"] Oct 30 21:41:54.134351 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.134315 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-envoy\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.134538 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.134386 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-credential-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.134538 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.134463 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-data\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.134538 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.134480 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8rpn\" (UniqueName: \"kubernetes.io/projected/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-kube-api-access-b8rpn\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.134538 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.134509 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-workload-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.134682 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.134651 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-token\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.134716 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.134695 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-podinfo\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.134761 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.134721 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-workload-certs\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.134845 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.134821 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istiod-ca-cert\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.151367 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.151333 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed945f27-nrnqc" event={"ID":"8a7b59b3-0b71-4359-8918-b0ed7a8d1496","Type":"ContainerStarted","Data":"9ef80080114e9814ff6fa12e462521682a6c7d788653e990a9d47b718ecfc1a3"} Oct 30 21:41:54.151549 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.151395 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed945f27-nrnqc" event={"ID":"8a7b59b3-0b71-4359-8918-b0ed7a8d1496","Type":"ContainerStarted","Data":"2cc0ca06b1c89cb66fea6df9d89db8b81a891512be3ff70e03f228a88c990b7b"} Oct 30 21:41:54.152829 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.152809 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d987ad-tt9nb" Oct 30 21:41:54.152923 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.152812 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d987ad-tt9nb" event={"ID":"7d5acfa7-3679-4ddf-91d4-db8d650f84b3","Type":"ContainerDied","Data":"9570901d6541fc526e282430cf68836db55d1c0ba5f8e13c4b2f4ce44035f955"} Oct 30 21:41:54.152923 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.152921 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9570901d6541fc526e282430cf68836db55d1c0ba5f8e13c4b2f4ce44035f955" Oct 30 21:41:54.154847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.154818 5801 generic.go:356] "Generic (PLEG): container finished" podID="f264c6aa-54c1-41cd-9012-334aef4c85f2" containerID="99592a71e1a630006caeb084d4ae81189497a4cb047dc13253f7fe9d15be2199" exitCode=0 Oct 30 21:41:54.155000 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.154878 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3003efe5-ppb5f" event={"ID":"f264c6aa-54c1-41cd-9012-334aef4c85f2","Type":"ContainerDied","Data":"99592a71e1a630006caeb084d4ae81189497a4cb047dc13253f7fe9d15be2199"} Oct 30 21:41:54.171155 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.171093 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed945f27-nrnqc" podStartSLOduration=2.171078629 podStartE2EDuration="2.171078629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:54.169638814 +0000 UTC m=+782.765455464" watchObservedRunningTime="2026-10-30 21:41:54.171078629 +0000 UTC m=+782.766895289" Oct 30 21:41:54.236049 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.236010 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-token\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.236242 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.236065 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-podinfo\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.236242 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.236096 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-workload-certs\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.236242 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.236120 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istiod-ca-cert\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.236242 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.236228 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-envoy\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.236433 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.236301 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-credential-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.236433 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.236366 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-data\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.236433 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.236394 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8rpn\" (UniqueName: \"kubernetes.io/projected/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-kube-api-access-b8rpn\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.236433 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.236422 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-workload-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.236732 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.236707 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-data\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.236830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.236735 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-credential-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.236896 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.236840 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-workload-socket\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.237112 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.237083 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-workload-certs\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.237198 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.237127 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istiod-ca-cert\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.238593 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.238565 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-envoy\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.238829 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.238812 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-podinfo\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.248204 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.248177 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-token\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.248320 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.248300 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8rpn\" (UniqueName: \"kubernetes.io/projected/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-kube-api-access-b8rpn\") pod \"e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.358090 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.358056 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:54.367063 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.367023 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:54.374852 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.374672 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn"] Oct 30 21:41:54.381460 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.381428 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.396831 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.395705 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn"] Oct 30 21:41:54.441016 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.440338 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-credential-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.441016 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.440375 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk5rw\" (UniqueName: \"kubernetes.io/projected/321186ea-7520-430c-825b-b8129b422d95-kube-api-access-nk5rw\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.441016 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.440458 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-workload-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.441016 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.440490 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/321186ea-7520-430c-825b-b8129b422d95-istio-token\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.441016 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.440518 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-istio-data\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.441016 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.440614 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321186ea-7520-430c-825b-b8129b422d95-istiod-ca-cert\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.441016 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.440728 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321186ea-7520-430c-825b-b8129b422d95-istio-podinfo\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.441016 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.440773 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-istio-envoy\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.441016 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.440801 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-workload-certs\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.555974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.548129 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-workload-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.555974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.548211 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/321186ea-7520-430c-825b-b8129b422d95-istio-token\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.555974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.548262 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-istio-data\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.555974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.548420 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321186ea-7520-430c-825b-b8129b422d95-istiod-ca-cert\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.555974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.548596 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321186ea-7520-430c-825b-b8129b422d95-istio-podinfo\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.555974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.548626 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-istio-envoy\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.555974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.548672 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-workload-certs\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.555974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.548968 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-credential-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.555974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.549000 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk5rw\" (UniqueName: \"kubernetes.io/projected/321186ea-7520-430c-825b-b8129b422d95-kube-api-access-nk5rw\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.555974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.549859 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-workload-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.555974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.550643 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-workload-certs\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.555974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.550918 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-credential-socket\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.555974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.551171 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-istio-data\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.555974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.551350 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321186ea-7520-430c-825b-b8129b422d95-istiod-ca-cert\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.566296 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.562948 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/321186ea-7520-430c-825b-b8129b422d95-istio-token\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.566296 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.563239 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk5rw\" (UniqueName: \"kubernetes.io/projected/321186ea-7520-430c-825b-b8129b422d95-kube-api-access-nk5rw\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.568629 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.568459 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-istio-envoy\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.568904 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.568883 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321186ea-7520-430c-825b-b8129b422d95-istio-podinfo\") pod \"e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.746270 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.746184 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:54.757909 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.757874 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc"] Oct 30 21:41:54.769095 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.769060 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:54.769180 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.769127 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:54.769180 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:54.769154 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:55.046564 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:41:55.046525 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321186ea_7520_430c_825b_b8129b422d95.slice/crio-98eae10d7ea186e38f155762da2539fda0da794649c49fa9f5e6aaebab8b8287 WatchSource:0}: Error finding container 98eae10d7ea186e38f155762da2539fda0da794649c49fa9f5e6aaebab8b8287: Status 404 returned error can't find the container with id 98eae10d7ea186e38f155762da2539fda0da794649c49fa9f5e6aaebab8b8287 Oct 30 21:41:55.048234 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.048210 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn"] Oct 30 21:41:55.049809 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.049734 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:55.049878 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.049825 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:55.049878 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.049861 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:55.167133 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.167092 5801 generic.go:356] "Generic (PLEG): container finished" podID="8a7b59b3-0b71-4359-8918-b0ed7a8d1496" containerID="9ef80080114e9814ff6fa12e462521682a6c7d788653e990a9d47b718ecfc1a3" exitCode=0 Oct 30 21:41:55.167301 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.167184 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed945f27-nrnqc" event={"ID":"8a7b59b3-0b71-4359-8918-b0ed7a8d1496","Type":"ContainerDied","Data":"9ef80080114e9814ff6fa12e462521682a6c7d788653e990a9d47b718ecfc1a3"} Oct 30 21:41:55.169364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.169337 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" event={"ID":"321186ea-7520-430c-825b-b8129b422d95","Type":"ContainerStarted","Data":"470c2ceb080097ee36a8e3448c13b630cc6f4d2eb0fd654fa88e919cae426264"} Oct 30 21:41:55.169503 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.169381 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" event={"ID":"321186ea-7520-430c-825b-b8129b422d95","Type":"ContainerStarted","Data":"98eae10d7ea186e38f155762da2539fda0da794649c49fa9f5e6aaebab8b8287"} Oct 30 21:41:55.171776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.171428 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" event={"ID":"07fab6bc-5ee9-4724-aa10-dcf0df1af13b","Type":"ContainerStarted","Data":"a19a452c99d9201e48336b7e0e807f5096f517fe29e1f9c6137146695c1bcaee"} Oct 30 21:41:55.171776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.171492 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" event={"ID":"07fab6bc-5ee9-4724-aa10-dcf0df1af13b","Type":"ContainerStarted","Data":"4392d179f842f247143603d2e37e0a99d3d4812aef88cd4dd53863d12c29c5a0"} Oct 30 21:41:55.215015 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.214982 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:55.222765 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.222683 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" podStartSLOduration=1.222668068 podStartE2EDuration="1.222668068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:55.216701299 +0000 UTC m=+783.812517985" watchObservedRunningTime="2026-10-30 21:41:55.222668068 +0000 UTC m=+783.818484727" Oct 30 21:41:55.278317 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.277802 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" podStartSLOduration=2.277778921 podStartE2EDuration="2.277778921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:55.242064799 +0000 UTC m=+783.837881459" watchObservedRunningTime="2026-10-30 21:41:55.277778921 +0000 UTC m=+783.873595581" Oct 30 21:41:55.377518 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.372381 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:41:55.377518 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.372445 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" podUID="07fab6bc-5ee9-4724-aa10-dcf0df1af13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 21:41:55.377518 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.372723 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d08887a-ckqgm"] Oct 30 21:41:55.381948 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.378721 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:41:55.381948 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.378850 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d08887a-ckqgm" Oct 30 21:41:55.392729 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.392684 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d08887a-ckqgm"] Oct 30 21:41:55.489777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.487647 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbg6x\" (UniqueName: \"kubernetes.io/projected/3f2f31c2-d994-43c8-9213-040859e880ff-kube-api-access-kbg6x\") pod \"maas-api-revoke-sub-1d08887a-ckqgm\" (UID: \"3f2f31c2-d994-43c8-9213-040859e880ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d08887a-ckqgm" Oct 30 21:41:55.489777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.487731 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f2f31c2-d994-43c8-9213-040859e880ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d08887a-ckqgm\" (UID: \"3f2f31c2-d994-43c8-9213-040859e880ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d08887a-ckqgm" Oct 30 21:41:55.489777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.487912 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f2f31c2-d994-43c8-9213-040859e880ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d08887a-ckqgm\" (UID: \"3f2f31c2-d994-43c8-9213-040859e880ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d08887a-ckqgm" Oct 30 21:41:55.541785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.541744 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3003efe5-ppb5f" Oct 30 21:41:55.591317 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.591261 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f264c6aa-54c1-41cd-9012-334aef4c85f2-kube-api-access-8l9x4\" (UniqueName: \"kubernetes.io/projected/f264c6aa-54c1-41cd-9012-334aef4c85f2-kube-api-access-8l9x4\") pod \"f264c6aa-54c1-41cd-9012-334aef4c85f2\" (UID: \"f264c6aa-54c1-41cd-9012-334aef4c85f2\") " Oct 30 21:41:55.591876 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.591559 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f264c6aa-54c1-41cd-9012-334aef4c85f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f264c6aa-54c1-41cd-9012-334aef4c85f2-maas-api-xks-ca\") pod \"f264c6aa-54c1-41cd-9012-334aef4c85f2\" (UID: \"f264c6aa-54c1-41cd-9012-334aef4c85f2\") " Oct 30 21:41:55.591876 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.591679 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f264c6aa-54c1-41cd-9012-334aef4c85f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f264c6aa-54c1-41cd-9012-334aef4c85f2-maas-api-openshift-ca\") pod \"f264c6aa-54c1-41cd-9012-334aef4c85f2\" (UID: \"f264c6aa-54c1-41cd-9012-334aef4c85f2\") " Oct 30 21:41:55.592294 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.592266 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbg6x\" (UniqueName: \"kubernetes.io/projected/3f2f31c2-d994-43c8-9213-040859e880ff-kube-api-access-kbg6x\") pod \"maas-api-revoke-sub-1d08887a-ckqgm\" (UID: \"3f2f31c2-d994-43c8-9213-040859e880ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d08887a-ckqgm" Oct 30 21:41:55.592387 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.592354 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f2f31c2-d994-43c8-9213-040859e880ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d08887a-ckqgm\" (UID: \"3f2f31c2-d994-43c8-9213-040859e880ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d08887a-ckqgm" Oct 30 21:41:55.592547 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.592528 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f2f31c2-d994-43c8-9213-040859e880ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d08887a-ckqgm\" (UID: \"3f2f31c2-d994-43c8-9213-040859e880ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d08887a-ckqgm" Oct 30 21:41:55.594251 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.594204 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f2f31c2-d994-43c8-9213-040859e880ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d08887a-ckqgm\" (UID: \"3f2f31c2-d994-43c8-9213-040859e880ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d08887a-ckqgm" Oct 30 21:41:55.594381 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.594255 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f264c6aa-54c1-41cd-9012-334aef4c85f2-maas-api-openshift-ca" pod "f264c6aa-54c1-41cd-9012-334aef4c85f2" (UID: "f264c6aa-54c1-41cd-9012-334aef4c85f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:55.595508 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.595477 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f264c6aa-54c1-41cd-9012-334aef4c85f2-kube-api-access-8l9x4" pod "f264c6aa-54c1-41cd-9012-334aef4c85f2" (UID: "f264c6aa-54c1-41cd-9012-334aef4c85f2"). InnerVolumeSpecName "kube-api-access-8l9x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:55.597050 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.597020 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f2f31c2-d994-43c8-9213-040859e880ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d08887a-ckqgm\" (UID: \"3f2f31c2-d994-43c8-9213-040859e880ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d08887a-ckqgm" Oct 30 21:41:55.597144 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.597127 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f264c6aa-54c1-41cd-9012-334aef4c85f2-maas-api-xks-ca" pod "f264c6aa-54c1-41cd-9012-334aef4c85f2" (UID: "f264c6aa-54c1-41cd-9012-334aef4c85f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:55.610607 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.606781 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbg6x\" (UniqueName: \"kubernetes.io/projected/3f2f31c2-d994-43c8-9213-040859e880ff-kube-api-access-kbg6x\") pod \"maas-api-revoke-sub-1d08887a-ckqgm\" (UID: \"3f2f31c2-d994-43c8-9213-040859e880ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d08887a-ckqgm" Oct 30 21:41:55.694168 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.694079 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f264c6aa-54c1-41cd-9012-334aef4c85f2-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:55.694168 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.694110 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l9x4\" (UniqueName: \"kubernetes.io/projected/f264c6aa-54c1-41cd-9012-334aef4c85f2-kube-api-access-8l9x4\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:55.694168 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.694124 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f264c6aa-54c1-41cd-9012-334aef4c85f2-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:55.725529 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.725500 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d08887a-ckqgm" Oct 30 21:41:55.746796 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:55.746766 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:41:56.038280 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:41:56.038249 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2f31c2_d994_43c8_9213_040859e880ff.slice/crio-fcb467fd39a1e6d9d862ac30f647f95fd48ccab2c9260bea2b84e392d940261f WatchSource:0}: Error finding container fcb467fd39a1e6d9d862ac30f647f95fd48ccab2c9260bea2b84e392d940261f: Status 404 returned error can't find the container with id fcb467fd39a1e6d9d862ac30f647f95fd48ccab2c9260bea2b84e392d940261f Oct 30 21:41:56.039696 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:56.039666 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d08887a-ckqgm"] Oct 30 21:41:56.209272 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:56.209229 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d08887a-ckqgm" event={"ID":"3f2f31c2-d994-43c8-9213-040859e880ff","Type":"ContainerStarted","Data":"fcb467fd39a1e6d9d862ac30f647f95fd48ccab2c9260bea2b84e392d940261f"} Oct 30 21:41:56.211431 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:56.211404 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3003efe5-ppb5f" Oct 30 21:41:56.211568 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:56.211470 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3003efe5-ppb5f" event={"ID":"f264c6aa-54c1-41cd-9012-334aef4c85f2","Type":"ContainerDied","Data":"23b87253af31471edd03f694ec0908d28a850f8e485e23513ecbb184e54933c5"} Oct 30 21:41:56.211568 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:56.211531 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23b87253af31471edd03f694ec0908d28a850f8e485e23513ecbb184e54933c5" Oct 30 21:41:56.294832 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:56.294583 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:56.379775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:56.376789 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:41:56.379775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:56.376856 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" podUID="07fab6bc-5ee9-4724-aa10-dcf0df1af13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 21:41:56.713020 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:56.712991 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed945f27-nrnqc" Oct 30 21:41:56.744861 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:56.744352 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a7b59b3-0b71-4359-8918-b0ed7a8d1496-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a7b59b3-0b71-4359-8918-b0ed7a8d1496-maas-api-openshift-ca\") pod \"8a7b59b3-0b71-4359-8918-b0ed7a8d1496\" (UID: \"8a7b59b3-0b71-4359-8918-b0ed7a8d1496\") " Oct 30 21:41:56.744861 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:56.744531 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a7b59b3-0b71-4359-8918-b0ed7a8d1496-kube-api-access-k9lrp\" (UniqueName: \"kubernetes.io/projected/8a7b59b3-0b71-4359-8918-b0ed7a8d1496-kube-api-access-k9lrp\") pod \"8a7b59b3-0b71-4359-8918-b0ed7a8d1496\" (UID: \"8a7b59b3-0b71-4359-8918-b0ed7a8d1496\") " Oct 30 21:41:56.744861 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:56.744565 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a7b59b3-0b71-4359-8918-b0ed7a8d1496-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a7b59b3-0b71-4359-8918-b0ed7a8d1496-maas-api-xks-ca\") pod \"8a7b59b3-0b71-4359-8918-b0ed7a8d1496\" (UID: \"8a7b59b3-0b71-4359-8918-b0ed7a8d1496\") " Oct 30 21:41:56.745466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:56.745376 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7b59b3-0b71-4359-8918-b0ed7a8d1496-maas-api-openshift-ca" pod "8a7b59b3-0b71-4359-8918-b0ed7a8d1496" (UID: "8a7b59b3-0b71-4359-8918-b0ed7a8d1496"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:56.745637 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:56.745618 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a7b59b3-0b71-4359-8918-b0ed7a8d1496-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:56.747389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:56.747359 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7b59b3-0b71-4359-8918-b0ed7a8d1496-kube-api-access-k9lrp" pod "8a7b59b3-0b71-4359-8918-b0ed7a8d1496" (UID: "8a7b59b3-0b71-4359-8918-b0ed7a8d1496"). InnerVolumeSpecName "kube-api-access-k9lrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:56.748014 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:56.747986 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:41:56.748076 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:56.748043 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" podUID="321186ea-7520-430c-825b-b8129b422d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:41:56.748599 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:56.748579 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a7b59b3-0b71-4359-8918-b0ed7a8d1496-maas-api-xks-ca" pod "8a7b59b3-0b71-4359-8918-b0ed7a8d1496" (UID: "8a7b59b3-0b71-4359-8918-b0ed7a8d1496"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:56.847472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:56.847429 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9lrp\" (UniqueName: \"kubernetes.io/projected/8a7b59b3-0b71-4359-8918-b0ed7a8d1496-kube-api-access-k9lrp\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:56.847472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:56.847477 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a7b59b3-0b71-4359-8918-b0ed7a8d1496-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:57.224332 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:57.224280 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d08887a-ckqgm" event={"ID":"3f2f31c2-d994-43c8-9213-040859e880ff","Type":"ContainerStarted","Data":"723cbdb1ceb6d6f72257e39e51165a49c37e4cd9dc8c6f6e346a05736e37aa9d"} Oct 30 21:41:57.226006 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:57.225979 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed945f27-nrnqc" event={"ID":"8a7b59b3-0b71-4359-8918-b0ed7a8d1496","Type":"ContainerDied","Data":"2cc0ca06b1c89cb66fea6df9d89db8b81a891512be3ff70e03f228a88c990b7b"} Oct 30 21:41:57.226006 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:57.225997 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed945f27-nrnqc" Oct 30 21:41:57.226182 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:57.226027 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cc0ca06b1c89cb66fea6df9d89db8b81a891512be3ff70e03f228a88c990b7b" Oct 30 21:41:57.250470 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:57.250403 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d08887a-ckqgm" podStartSLOduration=2.25038263 podStartE2EDuration="2.25038263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:57.242656093 +0000 UTC m=+785.838472756" watchObservedRunningTime="2026-10-30 21:41:57.25038263 +0000 UTC m=+785.846199291" Oct 30 21:41:57.400774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:57.400403 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:57.403386 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:57.402361 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:41:57.403386 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:57.402415 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" podUID="07fab6bc-5ee9-4724-aa10-dcf0df1af13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 21:41:57.511819 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:57.510784 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:57.746821 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:57.746780 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:41:57.747212 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:57.746844 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" podUID="321186ea-7520-430c-825b-b8129b422d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:41:58.235922 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:58.235894 5801 generic.go:356] "Generic (PLEG): container finished" podID="3f2f31c2-d994-43c8-9213-040859e880ff" containerID="723cbdb1ceb6d6f72257e39e51165a49c37e4cd9dc8c6f6e346a05736e37aa9d" exitCode=0 Oct 30 21:41:58.236190 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:58.235971 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d08887a-ckqgm" event={"ID":"3f2f31c2-d994-43c8-9213-040859e880ff","Type":"ContainerDied","Data":"723cbdb1ceb6d6f72257e39e51165a49c37e4cd9dc8c6f6e346a05736e37aa9d"} Oct 30 21:41:58.375777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:58.368239 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:41:58.375777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:58.368302 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" podUID="07fab6bc-5ee9-4724-aa10-dcf0df1af13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 21:41:58.747587 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:58.747552 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:41:58.748021 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:58.747603 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" podUID="321186ea-7520-430c-825b-b8129b422d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:41:58.770776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:58.764381 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:59.376457 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:59.376395 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:41:59.376669 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:59.376462 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" podUID="07fab6bc-5ee9-4724-aa10-dcf0df1af13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 21:41:59.616655 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:59.616629 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d08887a-ckqgm" Oct 30 21:41:59.636179 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:59.636144 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f2f31c2-d994-43c8-9213-040859e880ff-kube-api-access-kbg6x\" (UniqueName: \"kubernetes.io/projected/3f2f31c2-d994-43c8-9213-040859e880ff-kube-api-access-kbg6x\") pod \"3f2f31c2-d994-43c8-9213-040859e880ff\" (UID: \"3f2f31c2-d994-43c8-9213-040859e880ff\") " Oct 30 21:41:59.636935 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:59.636913 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f2f31c2-d994-43c8-9213-040859e880ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f2f31c2-d994-43c8-9213-040859e880ff-maas-api-openshift-ca\") pod \"3f2f31c2-d994-43c8-9213-040859e880ff\" (UID: \"3f2f31c2-d994-43c8-9213-040859e880ff\") " Oct 30 21:41:59.637166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:59.637148 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f2f31c2-d994-43c8-9213-040859e880ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f2f31c2-d994-43c8-9213-040859e880ff-maas-api-xks-ca\") pod \"3f2f31c2-d994-43c8-9213-040859e880ff\" (UID: \"3f2f31c2-d994-43c8-9213-040859e880ff\") " Oct 30 21:41:59.637313 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:59.637286 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f2f31c2-d994-43c8-9213-040859e880ff-maas-api-openshift-ca" pod "3f2f31c2-d994-43c8-9213-040859e880ff" (UID: "3f2f31c2-d994-43c8-9213-040859e880ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:59.638037 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:59.638016 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f2f31c2-d994-43c8-9213-040859e880ff-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:59.639974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:59.639947 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f2f31c2-d994-43c8-9213-040859e880ff-maas-api-xks-ca" pod "3f2f31c2-d994-43c8-9213-040859e880ff" (UID: "3f2f31c2-d994-43c8-9213-040859e880ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:59.640379 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:59.640343 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2f31c2-d994-43c8-9213-040859e880ff-kube-api-access-kbg6x" pod "3f2f31c2-d994-43c8-9213-040859e880ff" (UID: "3f2f31c2-d994-43c8-9213-040859e880ff"). InnerVolumeSpecName "kube-api-access-kbg6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:59.740641 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:59.740600 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbg6x\" (UniqueName: \"kubernetes.io/projected/3f2f31c2-d994-43c8-9213-040859e880ff-kube-api-access-kbg6x\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:59.740641 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:59.740644 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f2f31c2-d994-43c8-9213-040859e880ff-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:59.747689 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:59.747652 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:41:59.748036 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:59.747717 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" podUID="321186ea-7520-430c-825b-b8129b422d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:41:59.817801 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:41:59.817678 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:00.316777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:00.310350 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d08887a-ckqgm" event={"ID":"3f2f31c2-d994-43c8-9213-040859e880ff","Type":"ContainerDied","Data":"fcb467fd39a1e6d9d862ac30f647f95fd48ccab2c9260bea2b84e392d940261f"} Oct 30 21:42:00.316777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:00.310416 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcb467fd39a1e6d9d862ac30f647f95fd48ccab2c9260bea2b84e392d940261f" Oct 30 21:42:00.316777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:00.310574 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d08887a-ckqgm" Oct 30 21:42:00.368226 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:00.368181 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:42:00.368437 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:00.368253 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" podUID="07fab6bc-5ee9-4724-aa10-dcf0df1af13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 21:42:00.656829 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:42:00.599988 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:42:00.752776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:00.746606 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:42:00.752776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:00.746665 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" podUID="321186ea-7520-430c-825b-b8129b422d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:42:01.376780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:01.368341 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:42:01.376780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:01.368437 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" podUID="07fab6bc-5ee9-4724-aa10-dcf0df1af13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 21:42:01.473777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:01.467577 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:42:01.473777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:01.467885 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:42:01.558778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:01.554624 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:01.639837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:01.629826 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:42:01.939111 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:01.939020 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:42:02.030773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:01.980976 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:42:02.055780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:02.020440 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:02.055780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:02.030565 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" podUID="321186ea-7520-430c-825b-b8129b422d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:42:02.101790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:02.099026 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:02.205376 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:02.205276 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:02.226076 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:02.226032 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:02.248105 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:02.247846 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:02.309729 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:02.309685 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:02.340242 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:02.339834 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:02.383712 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:02.383666 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:42:02.406271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:02.406215 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" podUID="07fab6bc-5ee9-4724-aa10-dcf0df1af13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 21:42:02.489522 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:02.489413 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:02.514854 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:02.491831 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:02.515575 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:02.515539 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:42:02.516359 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:02.516342 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:42:02.595777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:02.595013 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:42:02.595777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:02.595578 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:42:02.623783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:02.623721 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:02.771976 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:02.667825 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:42:02.829536 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:02.772694 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:42:02.829536 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:02.804852 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:42:02.829536 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:02.823915 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" podUID="321186ea-7520-430c-825b-b8129b422d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:42:02.979951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:02.979895 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:03.018437 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:02.982646 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:03.051403 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:02.982797 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:42:03.143010 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:03.090442 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:03.202614 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:03.091078 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:42:03.203599 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:03.111731 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:03.249079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:03.236927 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:42:03.249079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:03.202909 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:42:03.258775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:03.251602 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:03.293778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:03.261988 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:42:03.328214 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:03.328183 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:42:03.330979 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:03.330953 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:42:03.697230 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:03.697145 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:42:03.725842 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:03.340074 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:42:03.741649 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:03.380022 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:42:03.758935 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:03.758889 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" podUID="07fab6bc-5ee9-4724-aa10-dcf0df1af13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 21:42:03.759235 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:03.560960 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:03.761268 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:03.502134 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:03.762504 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:03.630824 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:03.763623 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:03.379876 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:03.774912 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:03.774863 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:03.906778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:03.905093 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:42:03.906778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:03.905158 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" podUID="321186ea-7520-430c-825b-b8129b422d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:42:04.137786 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:04.134149 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:42:04.212799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:04.210892 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:04.212799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:04.211322 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:04.212799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:04.211595 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:42:04.269868 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:04.267580 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:42:04.269868 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:04.269564 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:42:04.389166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:04.389070 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 21:42:04.400785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:04.394118 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:42:04.422175 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:04.422138 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:42:04.429777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:04.425826 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" podUID="07fab6bc-5ee9-4724-aa10-dcf0df1af13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 21:42:04.508778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:04.439389 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:42:04.520779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:04.512890 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:42:04.541794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:04.528907 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:42:04.741968 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:04.741858 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:04.762918 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:04.762853 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:42:04.771785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:04.763518 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:05.057775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:04.773010 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:42:05.057775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:04.773224 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:05.057775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:04.776468 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:42:05.057775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:05.056933 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" podUID="321186ea-7520-430c-825b-b8129b422d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:42:05.057775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:04.776509 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:05.241780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:05.216838 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:42:05.284195 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:05.284142 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded" Oct 30 21:42:05.304841 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:05.286837 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:42:05.304841 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:05.287676 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:42:05.335031 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:05.334923 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded" Oct 30 21:42:05.365943 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:05.365899 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:05.366576 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:05.366543 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:42:05.392143 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:05.392094 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:42:05.392901 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:05.392862 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Oct 30 21:42:05.479343 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:05.479286 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:05.602651 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:05.602549 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": context deadline exceeded" Oct 30 21:42:05.687158 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:05.667185 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:42:05.687158 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:05.667251 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" podUID="07fab6bc-5ee9-4724-aa10-dcf0df1af13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 21:42:05.761779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:05.757193 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:05.786778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:05.785705 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:42:05.786778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:05.785795 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" podUID="321186ea-7520-430c-825b-b8129b422d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:42:05.813314 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:05.813249 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:06.179145 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.134134 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:06.179145 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.134570 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:42:06.179145 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.134921 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:42:06.179145 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.135213 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:42:06.179145 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.154032 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:06.261824 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.245966 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:42:06.261824 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.249198 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:06.312290 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.312240 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:06.313851 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.313114 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:06.351504 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.351465 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:42:06.351979 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.351825 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" podUID="db7b6603-3dfd-4f20-a362-da44b2fb9da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:06.381906 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.381846 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:06.382539 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.382499 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:42:06.410883 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.410828 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:42:06.481837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.481700 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:06.555850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.554901 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:42:06.555850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.554970 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" podUID="b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:06.755188 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.755089 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:42:06.755438 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.755406 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" podUID="321186ea-7520-430c-825b-b8129b422d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:42:06.759865 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.759828 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:06.837031 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.836983 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:42:06.846778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.839334 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed2fc544-ln794"] Oct 30 21:42:06.846778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.840310 5801 cpu_manager.go:414] "containerMap: removing container" podUID="3f2f31c2-d994-43c8-9213-040859e880ff" containerName="revoke-keys" Oct 30 21:42:06.846778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.840330 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f2f31c2-d994-43c8-9213-040859e880ff" containerName="revoke-keys" Oct 30 21:42:06.846778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.840389 5801 cpu_manager.go:414] "containerMap: removing container" podUID="8a7b59b3-0b71-4359-8918-b0ed7a8d1496" containerName="revoke-keys" Oct 30 21:42:06.846778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.840398 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a7b59b3-0b71-4359-8918-b0ed7a8d1496" containerName="revoke-keys" Oct 30 21:42:06.846778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.840421 5801 cpu_manager.go:414] "containerMap: removing container" podUID="f264c6aa-54c1-41cd-9012-334aef4c85f2" containerName="revoke-keys" Oct 30 21:42:06.846778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.840429 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f264c6aa-54c1-41cd-9012-334aef4c85f2" containerName="revoke-keys" Oct 30 21:42:06.846778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.840583 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a7b59b3-0b71-4359-8918-b0ed7a8d1496" containerName="revoke-keys" Oct 30 21:42:06.846778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.840600 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="f264c6aa-54c1-41cd-9012-334aef4c85f2" containerName="revoke-keys" Oct 30 21:42:06.846778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.840613 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f2f31c2-d994-43c8-9213-040859e880ff" containerName="revoke-keys" Oct 30 21:42:06.849913 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.849889 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2fc544-ln794" Oct 30 21:42:06.875847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.875521 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:42:06.876262 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.876241 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:42:06.877826 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.877802 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:42:06.878076 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.878024 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:42:06.899861 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.878886 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:42:06.899861 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.896233 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" podUID="311939a8-e35d-4636-b052-a368f0a41a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:06.977009 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.976972 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed2fc544-ln794"] Oct 30 21:42:06.988042 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.988001 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07cdfb9-6011-4083-9743-8d3503646a95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed2fc544-ln794\" (UID: \"b07cdfb9-6011-4083-9743-8d3503646a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2fc544-ln794" Oct 30 21:42:06.988303 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.988281 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdqf8\" (UniqueName: \"kubernetes.io/projected/b07cdfb9-6011-4083-9743-8d3503646a95-kube-api-access-pdqf8\") pod \"maas-api-revoke-sub-ed2fc544-ln794\" (UID: \"b07cdfb9-6011-4083-9743-8d3503646a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2fc544-ln794" Oct 30 21:42:06.988490 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:06.988472 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07cdfb9-6011-4083-9743-8d3503646a95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed2fc544-ln794\" (UID: \"b07cdfb9-6011-4083-9743-8d3503646a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2fc544-ln794" Oct 30 21:42:07.143983 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:07.143921 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:07.161955 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:07.161894 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:07.204129 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:07.202830 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:42:07.204129 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:07.203261 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:42:07.225663 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:07.225628 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07cdfb9-6011-4083-9743-8d3503646a95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed2fc544-ln794\" (UID: \"b07cdfb9-6011-4083-9743-8d3503646a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2fc544-ln794" Oct 30 21:42:07.226102 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:07.226082 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdqf8\" (UniqueName: \"kubernetes.io/projected/b07cdfb9-6011-4083-9743-8d3503646a95-kube-api-access-pdqf8\") pod \"maas-api-revoke-sub-ed2fc544-ln794\" (UID: \"b07cdfb9-6011-4083-9743-8d3503646a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2fc544-ln794" Oct 30 21:42:07.263079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:07.263039 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07cdfb9-6011-4083-9743-8d3503646a95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed2fc544-ln794\" (UID: \"b07cdfb9-6011-4083-9743-8d3503646a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2fc544-ln794" Oct 30 21:42:07.264510 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:07.264483 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07cdfb9-6011-4083-9743-8d3503646a95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed2fc544-ln794\" (UID: \"b07cdfb9-6011-4083-9743-8d3503646a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2fc544-ln794" Oct 30 21:42:07.268908 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:07.268874 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:42:07.306001 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:07.305962 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07cdfb9-6011-4083-9743-8d3503646a95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed2fc544-ln794\" (UID: \"b07cdfb9-6011-4083-9743-8d3503646a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2fc544-ln794" Oct 30 21:42:07.371868 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:07.337567 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdqf8\" (UniqueName: \"kubernetes.io/projected/b07cdfb9-6011-4083-9743-8d3503646a95-kube-api-access-pdqf8\") pod \"maas-api-revoke-sub-ed2fc544-ln794\" (UID: \"b07cdfb9-6011-4083-9743-8d3503646a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2fc544-ln794" Oct 30 21:42:07.372149 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:07.353270 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:42:07.404268 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:07.353311 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:07.417110 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:07.417065 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:42:07.418596 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:07.417335 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:07.436320 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:07.417383 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:07.460272 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:07.460234 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:42:07.460550 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:07.460520 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" podUID="07fab6bc-5ee9-4724-aa10-dcf0df1af13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:07.484229 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:07.484181 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:07.579779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:07.576164 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2fc544-ln794" Oct 30 21:42:07.809773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:07.806900 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:07.861779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:07.836958 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:42:07.862107 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:07.837908 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:07.867279 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:07.867219 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" podUID="321186ea-7520-430c-825b-b8129b422d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:42:08.112837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.102007 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c0755d-fgfdv"] Oct 30 21:42:08.243779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.229881 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:08.243779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.230444 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:42:08.243779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.232121 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:08.243779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.232933 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:42:08.285778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.278105 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:08.286310 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.286091 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c0755d-fgfdv"] Oct 30 21:42:08.286310 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.286218 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c0755d-fgfdv" Oct 30 21:42:08.413846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.404841 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a913f7e1-ae52-414b-a1fe-be53429bcf11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13c0755d-fgfdv\" (UID: \"a913f7e1-ae52-414b-a1fe-be53429bcf11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c0755d-fgfdv" Oct 30 21:42:08.413846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.404979 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpjfk\" (UniqueName: \"kubernetes.io/projected/a913f7e1-ae52-414b-a1fe-be53429bcf11-kube-api-access-qpjfk\") pod \"maas-api-revoke-sub-13c0755d-fgfdv\" (UID: \"a913f7e1-ae52-414b-a1fe-be53429bcf11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c0755d-fgfdv" Oct 30 21:42:08.413846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.405309 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a913f7e1-ae52-414b-a1fe-be53429bcf11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13c0755d-fgfdv\" (UID: \"a913f7e1-ae52-414b-a1fe-be53429bcf11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c0755d-fgfdv" Oct 30 21:42:08.413846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.405504 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:08.413846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.409584 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:08.442776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.433846 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:42:08.442776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.438834 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:08.442776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.439240 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:42:08.485978 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.484974 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 21:42:08.501406 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.500973 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sk2kw" podUID="faea70fa-2fa1-45b8-9409-72fb5d589cc3" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.25:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:08.509746 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.509702 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a913f7e1-ae52-414b-a1fe-be53429bcf11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13c0755d-fgfdv\" (UID: \"a913f7e1-ae52-414b-a1fe-be53429bcf11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c0755d-fgfdv" Oct 30 21:42:08.510099 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.510059 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a913f7e1-ae52-414b-a1fe-be53429bcf11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13c0755d-fgfdv\" (UID: \"a913f7e1-ae52-414b-a1fe-be53429bcf11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c0755d-fgfdv" Oct 30 21:42:08.510352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.510332 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpjfk\" (UniqueName: \"kubernetes.io/projected/a913f7e1-ae52-414b-a1fe-be53429bcf11-kube-api-access-qpjfk\") pod \"maas-api-revoke-sub-13c0755d-fgfdv\" (UID: \"a913f7e1-ae52-414b-a1fe-be53429bcf11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c0755d-fgfdv" Oct 30 21:42:08.511013 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.510933 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:42:08.522159 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.514827 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a913f7e1-ae52-414b-a1fe-be53429bcf11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13c0755d-fgfdv\" (UID: \"a913f7e1-ae52-414b-a1fe-be53429bcf11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c0755d-fgfdv" Oct 30 21:42:08.545779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.510983 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" podUID="07fab6bc-5ee9-4724-aa10-dcf0df1af13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:08.545779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.526163 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sk2kw" podUID="faea70fa-2fa1-45b8-9409-72fb5d589cc3" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.25:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:08.545779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.528541 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a913f7e1-ae52-414b-a1fe-be53429bcf11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13c0755d-fgfdv\" (UID: \"a913f7e1-ae52-414b-a1fe-be53429bcf11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c0755d-fgfdv" Oct 30 21:42:08.549776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.547743 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:08.587776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.562940 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpjfk\" (UniqueName: \"kubernetes.io/projected/a913f7e1-ae52-414b-a1fe-be53429bcf11-kube-api-access-qpjfk\") pod \"maas-api-revoke-sub-13c0755d-fgfdv\" (UID: \"a913f7e1-ae52-414b-a1fe-be53429bcf11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c0755d-fgfdv" Oct 30 21:42:08.587776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.578836 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:08.680400 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.680038 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c0755d-fgfdv" Oct 30 21:42:08.747229 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.747190 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:42:08.747512 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.747479 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" podUID="321186ea-7520-430c-825b-b8129b422d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:42:08.807779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.803938 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:08.813521 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.809364 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:08.928587 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:08.869869 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:09.230778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:09.230046 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:42:09.277837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:09.253047 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:42:09.277837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:09.253522 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:09.277837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:09.253803 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:09.277837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:09.275028 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:09.287174 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:09.278890 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:42:09.596849 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:09.411511 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:42:09.596849 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:09.414018 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:09.596849 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:09.471691 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:42:09.596849 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:09.495119 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:09.596849 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:09.516928 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:42:09.596849 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:09.518536 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:09.596849 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:09.542967 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:42:09.641055 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:09.641018 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:42:09.760779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:09.752515 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" podUID="07fab6bc-5ee9-4724-aa10-dcf0df1af13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:09.869364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:09.831250 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:42:09.963057 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:09.941730 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:42:10.207947 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:10.207912 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed2fc544-ln794"] Oct 30 21:42:10.209861 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:09.997094 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:42:10.248431 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:10.248384 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:42:10.270778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:10.264984 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:10.271967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:10.271567 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:42:10.314780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:10.298457 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:10.354938 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:10.289468 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:10.460928 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:10.431856 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:11.288210 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:10.541901 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:42:11.288919 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:10.567857 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:11.289158 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:10.568644 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:42:11.290825 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:10.568707 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:11.291071 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:10.734150 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:42:11.299655 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:10.781937 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:42:11.299902 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.299872 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" podUID="07fab6bc-5ee9-4724-aa10-dcf0df1af13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:11.300077 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.020078 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:11.300283 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.020114 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:42:11.302811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.091465 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2fc544-ln794" event={"ID":"b07cdfb9-6011-4083-9743-8d3503646a95","Type":"ContainerStarted","Data":"2cdb3b1210b704068d8d3cd6c358a0439d6b9c703a33d73899d368aa866146c4"} Oct 30 21:42:11.302983 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.222305 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:42:11.303170 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.092455 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:11.304265 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.270884 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:11.317911 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.317867 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:11.318613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.253890 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:11.319031 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.287744 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:11.361777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.347464 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:42:11.362163 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.362122 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" podUID="321186ea-7520-430c-825b-b8129b422d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:11.382231 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.378927 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:11.490607 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.490579 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:42:11.518302 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.518250 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c0755d-fgfdv"] Oct 30 21:42:11.579936 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.579908 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:42:11.580139 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.580082 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:42:11.651436 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.651403 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:42:11.651626 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.651459 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:42:11.651626 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.651484 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:42:11.651626 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.651514 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:42:11.651626 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.651537 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:42:11.748793 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.747321 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:42:11.792822 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.777767 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:11.826030 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.822892 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:42:11.826030 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.824267 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:42:11.832782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.829282 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:42:11.841702 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.841679 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:42:11.843136 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.843116 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:42:11.844391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:11.844367 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:42:12.112289 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:12.112210 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c0755d-fgfdv" event={"ID":"a913f7e1-ae52-414b-a1fe-be53429bcf11","Type":"ContainerStarted","Data":"17984719d8c6570ae4bb71ffc0e72ab37d3b640a36ffdf9876c9dc9fdce02a42"} Oct 30 21:42:12.112512 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:12.112344 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c0755d-fgfdv" event={"ID":"a913f7e1-ae52-414b-a1fe-be53429bcf11","Type":"ContainerStarted","Data":"3a82f1ae996842392744dd6caa3654959e6b720e7d8c4cdeb1c7dafa39f2caf6"} Oct 30 21:42:12.117088 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:12.117050 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2fc544-ln794" event={"ID":"b07cdfb9-6011-4083-9743-8d3503646a95","Type":"ContainerStarted","Data":"eda360ac88b172b2e2c99400252e3c3437360a680905ebb9656a76b083537a77"} Oct 30 21:42:12.146487 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:12.146408 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c0755d-fgfdv" podStartSLOduration=5.146386202 podStartE2EDuration="5.146386202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:12.133022145 +0000 UTC m=+800.728838806" watchObservedRunningTime="2026-10-30 21:42:12.146386202 +0000 UTC m=+800.742202864" Oct 30 21:42:12.158994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:12.158887 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2fc544-ln794" podStartSLOduration=8.158867755 podStartE2EDuration="8.158867755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:12.156557814 +0000 UTC m=+800.752374537" watchObservedRunningTime="2026-10-30 21:42:12.158867755 +0000 UTC m=+800.754684416" Oct 30 21:42:12.342931 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:12.342855 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:42:12.343354 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:12.342982 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" podUID="07fab6bc-5ee9-4724-aa10-dcf0df1af13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:12.736504 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:12.736456 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:12.781388 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:12.780033 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:42:12.787189 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:12.786984 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:42:13.127173 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:13.127143 5801 generic.go:356] "Generic (PLEG): container finished" podID="b07cdfb9-6011-4083-9743-8d3503646a95" containerID="eda360ac88b172b2e2c99400252e3c3437360a680905ebb9656a76b083537a77" exitCode=0 Oct 30 21:42:13.127334 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:13.127234 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2fc544-ln794" event={"ID":"b07cdfb9-6011-4083-9743-8d3503646a95","Type":"ContainerDied","Data":"eda360ac88b172b2e2c99400252e3c3437360a680905ebb9656a76b083537a77"} Oct 30 21:42:13.129102 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:13.129077 5801 generic.go:356] "Generic (PLEG): container finished" podID="a913f7e1-ae52-414b-a1fe-be53429bcf11" containerID="17984719d8c6570ae4bb71ffc0e72ab37d3b640a36ffdf9876c9dc9fdce02a42" exitCode=0 Oct 30 21:42:13.129228 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:13.129131 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c0755d-fgfdv" event={"ID":"a913f7e1-ae52-414b-a1fe-be53429bcf11","Type":"ContainerDied","Data":"17984719d8c6570ae4bb71ffc0e72ab37d3b640a36ffdf9876c9dc9fdce02a42"} Oct 30 21:42:13.544216 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:13.544136 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:14.246989 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:14.246959 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:14.362907 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:14.362879 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2fc544-ln794" Oct 30 21:42:14.414003 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:14.413930 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c0755d-fgfdv" Oct 30 21:42:14.472536 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:14.472501 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b07cdfb9-6011-4083-9743-8d3503646a95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07cdfb9-6011-4083-9743-8d3503646a95-maas-api-xks-ca\") pod \"b07cdfb9-6011-4083-9743-8d3503646a95\" (UID: \"b07cdfb9-6011-4083-9743-8d3503646a95\") " Oct 30 21:42:14.472713 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:14.472550 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a913f7e1-ae52-414b-a1fe-be53429bcf11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a913f7e1-ae52-414b-a1fe-be53429bcf11-maas-api-openshift-ca\") pod \"a913f7e1-ae52-414b-a1fe-be53429bcf11\" (UID: \"a913f7e1-ae52-414b-a1fe-be53429bcf11\") " Oct 30 21:42:14.472713 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:14.472607 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a913f7e1-ae52-414b-a1fe-be53429bcf11-kube-api-access-qpjfk\" (UniqueName: \"kubernetes.io/projected/a913f7e1-ae52-414b-a1fe-be53429bcf11-kube-api-access-qpjfk\") pod \"a913f7e1-ae52-414b-a1fe-be53429bcf11\" (UID: \"a913f7e1-ae52-414b-a1fe-be53429bcf11\") " Oct 30 21:42:14.472860 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:14.472808 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b07cdfb9-6011-4083-9743-8d3503646a95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07cdfb9-6011-4083-9743-8d3503646a95-maas-api-openshift-ca\") pod \"b07cdfb9-6011-4083-9743-8d3503646a95\" (UID: \"b07cdfb9-6011-4083-9743-8d3503646a95\") " Oct 30 21:42:14.472964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:14.472945 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a913f7e1-ae52-414b-a1fe-be53429bcf11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a913f7e1-ae52-414b-a1fe-be53429bcf11-maas-api-xks-ca\") pod \"a913f7e1-ae52-414b-a1fe-be53429bcf11\" (UID: \"a913f7e1-ae52-414b-a1fe-be53429bcf11\") " Oct 30 21:42:14.473038 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:14.472995 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a913f7e1-ae52-414b-a1fe-be53429bcf11-maas-api-openshift-ca" pod "a913f7e1-ae52-414b-a1fe-be53429bcf11" (UID: "a913f7e1-ae52-414b-a1fe-be53429bcf11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:14.473720 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:14.473673 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b07cdfb9-6011-4083-9743-8d3503646a95-maas-api-openshift-ca" pod "b07cdfb9-6011-4083-9743-8d3503646a95" (UID: "b07cdfb9-6011-4083-9743-8d3503646a95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:14.474050 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:14.474027 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b07cdfb9-6011-4083-9743-8d3503646a95-kube-api-access-pdqf8\" (UniqueName: \"kubernetes.io/projected/b07cdfb9-6011-4083-9743-8d3503646a95-kube-api-access-pdqf8\") pod \"b07cdfb9-6011-4083-9743-8d3503646a95\" (UID: \"b07cdfb9-6011-4083-9743-8d3503646a95\") " Oct 30 21:42:14.474895 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:14.474869 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a913f7e1-ae52-414b-a1fe-be53429bcf11-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:14.475031 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:14.474898 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07cdfb9-6011-4083-9743-8d3503646a95-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:14.475161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:14.475144 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a913f7e1-ae52-414b-a1fe-be53429bcf11-kube-api-access-qpjfk" pod "a913f7e1-ae52-414b-a1fe-be53429bcf11" (UID: "a913f7e1-ae52-414b-a1fe-be53429bcf11"). InnerVolumeSpecName "kube-api-access-qpjfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:14.475323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:14.475288 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b07cdfb9-6011-4083-9743-8d3503646a95-maas-api-xks-ca" pod "b07cdfb9-6011-4083-9743-8d3503646a95" (UID: "b07cdfb9-6011-4083-9743-8d3503646a95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:42:14.475456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:14.475426 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a913f7e1-ae52-414b-a1fe-be53429bcf11-maas-api-xks-ca" pod "a913f7e1-ae52-414b-a1fe-be53429bcf11" (UID: "a913f7e1-ae52-414b-a1fe-be53429bcf11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:42:14.476675 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:14.476649 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b07cdfb9-6011-4083-9743-8d3503646a95-kube-api-access-pdqf8" pod "b07cdfb9-6011-4083-9743-8d3503646a95" (UID: "b07cdfb9-6011-4083-9743-8d3503646a95"). InnerVolumeSpecName "kube-api-access-pdqf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:14.576215 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:14.576187 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdqf8\" (UniqueName: \"kubernetes.io/projected/b07cdfb9-6011-4083-9743-8d3503646a95-kube-api-access-pdqf8\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:14.576215 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:14.576215 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07cdfb9-6011-4083-9743-8d3503646a95-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:14.576584 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:14.576226 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpjfk\" (UniqueName: \"kubernetes.io/projected/a913f7e1-ae52-414b-a1fe-be53429bcf11-kube-api-access-qpjfk\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:14.576584 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:14.576236 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a913f7e1-ae52-414b-a1fe-be53429bcf11-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:15.035783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:15.035727 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:15.149276 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:15.149238 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c0755d-fgfdv" Oct 30 21:42:15.149475 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:15.149280 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c0755d-fgfdv" event={"ID":"a913f7e1-ae52-414b-a1fe-be53429bcf11","Type":"ContainerDied","Data":"3a82f1ae996842392744dd6caa3654959e6b720e7d8c4cdeb1c7dafa39f2caf6"} Oct 30 21:42:15.149475 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:15.149337 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a82f1ae996842392744dd6caa3654959e6b720e7d8c4cdeb1c7dafa39f2caf6" Oct 30 21:42:15.151214 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:15.151193 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2fc544-ln794" Oct 30 21:42:15.151214 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:15.151201 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2fc544-ln794" event={"ID":"b07cdfb9-6011-4083-9743-8d3503646a95","Type":"ContainerDied","Data":"2cdb3b1210b704068d8d3cd6c358a0439d6b9c703a33d73899d368aa866146c4"} Oct 30 21:42:15.151405 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:15.151246 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cdb3b1210b704068d8d3cd6c358a0439d6b9c703a33d73899d368aa866146c4" Oct 30 21:42:15.730107 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:15.730070 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:16.500505 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:16.500475 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7351-51e6cb"] Oct 30 21:42:16.501113 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:16.501099 5801 cpu_manager.go:414] "containerMap: removing container" podUID="b07cdfb9-6011-4083-9743-8d3503646a95" containerName="revoke-keys" Oct 30 21:42:16.501166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:16.501115 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b07cdfb9-6011-4083-9743-8d3503646a95" containerName="revoke-keys" Oct 30 21:42:16.501166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:16.501139 5801 cpu_manager.go:414] "containerMap: removing container" podUID="a913f7e1-ae52-414b-a1fe-be53429bcf11" containerName="revoke-keys" Oct 30 21:42:16.501166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:16.501145 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a913f7e1-ae52-414b-a1fe-be53429bcf11" containerName="revoke-keys" Oct 30 21:42:16.501256 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:16.501252 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="a913f7e1-ae52-414b-a1fe-be53429bcf11" containerName="revoke-keys" Oct 30 21:42:16.501334 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:16.501266 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="b07cdfb9-6011-4083-9743-8d3503646a95" containerName="revoke-keys" Oct 30 21:42:16.506295 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:16.506273 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7351-51e6cb" Oct 30 21:42:16.510104 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:16.510065 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 21:42:16.510216 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:16.510199 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 21:42:16.511132 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:16.511115 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-l4j2n\"" Oct 30 21:42:16.512522 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:16.512501 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7351-51e6cb"] Oct 30 21:42:16.611823 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:16.611784 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz9qz\" (UniqueName: \"kubernetes.io/projected/486c5963-3aad-47de-b5de-23595e3abec1-kube-api-access-cz9qz\") pod \"test-curl-7351-51e6cb\" (UID: \"486c5963-3aad-47de-b5de-23595e3abec1\") " pod="opendatahub/test-curl-7351-51e6cb" Oct 30 21:42:16.711128 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:16.711100 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:16.713616 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:16.713588 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz9qz\" (UniqueName: \"kubernetes.io/projected/486c5963-3aad-47de-b5de-23595e3abec1-kube-api-access-cz9qz\") pod \"test-curl-7351-51e6cb\" (UID: \"486c5963-3aad-47de-b5de-23595e3abec1\") " pod="opendatahub/test-curl-7351-51e6cb" Oct 30 21:42:16.725599 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:16.725576 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz9qz\" (UniqueName: \"kubernetes.io/projected/486c5963-3aad-47de-b5de-23595e3abec1-kube-api-access-cz9qz\") pod \"test-curl-7351-51e6cb\" (UID: \"486c5963-3aad-47de-b5de-23595e3abec1\") " pod="opendatahub/test-curl-7351-51e6cb" Oct 30 21:42:16.821849 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:16.821822 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7351-51e6cb" Oct 30 21:42:17.129341 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:42:17.129313 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod486c5963_3aad_47de_b5de_23595e3abec1.slice/crio-f79445c64d2e433654cbf747bcb19336497d5ff54b1de7f045fd603e7c1f8171 WatchSource:0}: Error finding container f79445c64d2e433654cbf747bcb19336497d5ff54b1de7f045fd603e7c1f8171: Status 404 returned error can't find the container with id f79445c64d2e433654cbf747bcb19336497d5ff54b1de7f045fd603e7c1f8171 Oct 30 21:42:17.130955 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:17.130931 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7351-51e6cb"] Oct 30 21:42:17.171392 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:17.171349 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7351-51e6cb" event={"ID":"486c5963-3aad-47de-b5de-23595e3abec1","Type":"ContainerStarted","Data":"f79445c64d2e433654cbf747bcb19336497d5ff54b1de7f045fd603e7c1f8171"} Oct 30 21:42:17.475794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:17.475707 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81dc9aa0-4rk86"] Oct 30 21:42:17.481560 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:17.481538 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9aa0-4rk86" Oct 30 21:42:17.487733 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:17.487706 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:42:17.489320 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:17.488858 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:42:17.489320 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:17.488901 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:42:17.489320 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:17.488940 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:42:17.496813 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:17.491595 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81dc9aa0-4rk86"] Oct 30 21:42:17.592112 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:17.592078 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:17.628682 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:17.628653 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4600e16-207d-45f5-bbb4-55cda56ce625-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81dc9aa0-4rk86\" (UID: \"c4600e16-207d-45f5-bbb4-55cda56ce625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9aa0-4rk86" Oct 30 21:42:17.628878 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:17.628859 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq2px\" (UniqueName: \"kubernetes.io/projected/c4600e16-207d-45f5-bbb4-55cda56ce625-kube-api-access-wq2px\") pod \"maas-api-revoke-sub-81dc9aa0-4rk86\" (UID: \"c4600e16-207d-45f5-bbb4-55cda56ce625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9aa0-4rk86" Oct 30 21:42:17.629108 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:17.629088 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4600e16-207d-45f5-bbb4-55cda56ce625-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81dc9aa0-4rk86\" (UID: \"c4600e16-207d-45f5-bbb4-55cda56ce625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9aa0-4rk86" Oct 30 21:42:17.741827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:17.741710 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq2px\" (UniqueName: \"kubernetes.io/projected/c4600e16-207d-45f5-bbb4-55cda56ce625-kube-api-access-wq2px\") pod \"maas-api-revoke-sub-81dc9aa0-4rk86\" (UID: \"c4600e16-207d-45f5-bbb4-55cda56ce625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9aa0-4rk86" Oct 30 21:42:17.742041 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:17.742024 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4600e16-207d-45f5-bbb4-55cda56ce625-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81dc9aa0-4rk86\" (UID: \"c4600e16-207d-45f5-bbb4-55cda56ce625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9aa0-4rk86" Oct 30 21:42:17.742086 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:17.742072 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4600e16-207d-45f5-bbb4-55cda56ce625-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81dc9aa0-4rk86\" (UID: \"c4600e16-207d-45f5-bbb4-55cda56ce625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9aa0-4rk86" Oct 30 21:42:17.742825 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:17.742775 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4600e16-207d-45f5-bbb4-55cda56ce625-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81dc9aa0-4rk86\" (UID: \"c4600e16-207d-45f5-bbb4-55cda56ce625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9aa0-4rk86" Oct 30 21:42:17.744769 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:17.744731 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4600e16-207d-45f5-bbb4-55cda56ce625-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81dc9aa0-4rk86\" (UID: \"c4600e16-207d-45f5-bbb4-55cda56ce625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9aa0-4rk86" Oct 30 21:42:17.755025 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:17.755000 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq2px\" (UniqueName: \"kubernetes.io/projected/c4600e16-207d-45f5-bbb4-55cda56ce625-kube-api-access-wq2px\") pod \"maas-api-revoke-sub-81dc9aa0-4rk86\" (UID: \"c4600e16-207d-45f5-bbb4-55cda56ce625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9aa0-4rk86" Oct 30 21:42:17.802159 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:17.802131 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9aa0-4rk86" Oct 30 21:42:18.090447 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:42:18.090410 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4600e16_207d_45f5_bbb4_55cda56ce625.slice/crio-d9546c9b90204ca29d843666c36127e30adca46c69a58f5ad0fe3475a79c6596 WatchSource:0}: Error finding container d9546c9b90204ca29d843666c36127e30adca46c69a58f5ad0fe3475a79c6596: Status 404 returned error can't find the container with id d9546c9b90204ca29d843666c36127e30adca46c69a58f5ad0fe3475a79c6596 Oct 30 21:42:18.091508 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:18.091484 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81dc9aa0-4rk86"] Oct 30 21:42:18.180544 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:18.180515 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9aa0-4rk86" event={"ID":"c4600e16-207d-45f5-bbb4-55cda56ce625","Type":"ContainerStarted","Data":"d9546c9b90204ca29d843666c36127e30adca46c69a58f5ad0fe3475a79c6596"} Oct 30 21:42:18.510175 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:18.510135 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:19.192812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:19.192741 5801 generic.go:356] "Generic (PLEG): container finished" podID="c4600e16-207d-45f5-bbb4-55cda56ce625" containerID="d36dad163ff5cafe8843435eb296643c38324b5268d76b0aa884637759bf0b3f" exitCode=0 Oct 30 21:42:19.193222 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:19.192825 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9aa0-4rk86" event={"ID":"c4600e16-207d-45f5-bbb4-55cda56ce625","Type":"ContainerDied","Data":"d36dad163ff5cafe8843435eb296643c38324b5268d76b0aa884637759bf0b3f"} Oct 30 21:42:19.427480 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:19.427440 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:19.939555 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:19.939525 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd3352f7-vkjzp"] Oct 30 21:42:19.945589 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:19.945565 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3352f7-vkjzp" Oct 30 21:42:19.954702 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:19.954676 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd3352f7-vkjzp"] Oct 30 21:42:20.081553 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.081515 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwfst\" (UniqueName: \"kubernetes.io/projected/545f6262-2b94-4b51-9688-d08feb89cb59-kube-api-access-fwfst\") pod \"maas-api-revoke-sub-bd3352f7-vkjzp\" (UID: \"545f6262-2b94-4b51-9688-d08feb89cb59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3352f7-vkjzp" Oct 30 21:42:20.081711 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.081655 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545f6262-2b94-4b51-9688-d08feb89cb59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd3352f7-vkjzp\" (UID: \"545f6262-2b94-4b51-9688-d08feb89cb59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3352f7-vkjzp" Oct 30 21:42:20.081858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.081844 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545f6262-2b94-4b51-9688-d08feb89cb59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd3352f7-vkjzp\" (UID: \"545f6262-2b94-4b51-9688-d08feb89cb59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3352f7-vkjzp" Oct 30 21:42:20.183626 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.183544 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545f6262-2b94-4b51-9688-d08feb89cb59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd3352f7-vkjzp\" (UID: \"545f6262-2b94-4b51-9688-d08feb89cb59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3352f7-vkjzp" Oct 30 21:42:20.183779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.183686 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545f6262-2b94-4b51-9688-d08feb89cb59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd3352f7-vkjzp\" (UID: \"545f6262-2b94-4b51-9688-d08feb89cb59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3352f7-vkjzp" Oct 30 21:42:20.183862 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.183849 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwfst\" (UniqueName: \"kubernetes.io/projected/545f6262-2b94-4b51-9688-d08feb89cb59-kube-api-access-fwfst\") pod \"maas-api-revoke-sub-bd3352f7-vkjzp\" (UID: \"545f6262-2b94-4b51-9688-d08feb89cb59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3352f7-vkjzp" Oct 30 21:42:20.184308 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.184288 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545f6262-2b94-4b51-9688-d08feb89cb59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd3352f7-vkjzp\" (UID: \"545f6262-2b94-4b51-9688-d08feb89cb59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3352f7-vkjzp" Oct 30 21:42:20.186037 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.186014 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545f6262-2b94-4b51-9688-d08feb89cb59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd3352f7-vkjzp\" (UID: \"545f6262-2b94-4b51-9688-d08feb89cb59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3352f7-vkjzp" Oct 30 21:42:20.204740 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.204707 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7351-51e6cb" event={"ID":"486c5963-3aad-47de-b5de-23595e3abec1","Type":"ContainerStarted","Data":"b3caa783cab09026bbe8f6fa88438eaaaac9a80b22768ed71cada604068ed99e"} Oct 30 21:42:20.208462 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.208436 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwfst\" (UniqueName: \"kubernetes.io/projected/545f6262-2b94-4b51-9688-d08feb89cb59-kube-api-access-fwfst\") pod \"maas-api-revoke-sub-bd3352f7-vkjzp\" (UID: \"545f6262-2b94-4b51-9688-d08feb89cb59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3352f7-vkjzp" Oct 30 21:42:20.230901 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.230846 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7351-51e6cb" podStartSLOduration=1.467613332 podStartE2EDuration="4.230831396s" totalImagesPullingTime="2.763218064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:20.229178061 +0000 UTC m=+808.824994740" watchObservedRunningTime="2026-10-30 21:42:20.230831396 +0000 UTC m=+808.826648057" Oct 30 21:42:20.287811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.282340 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3352f7-vkjzp" Oct 30 21:42:20.376003 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.375971 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9aa0-4rk86" Oct 30 21:42:20.455310 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.455199 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:20.459607 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.459583 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a019e5ae-x55jc"] Oct 30 21:42:20.460312 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.460296 5801 cpu_manager.go:414] "containerMap: removing container" podUID="c4600e16-207d-45f5-bbb4-55cda56ce625" containerName="revoke-keys" Oct 30 21:42:20.460380 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.460314 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4600e16-207d-45f5-bbb4-55cda56ce625" containerName="revoke-keys" Oct 30 21:42:20.460456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.460445 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4600e16-207d-45f5-bbb4-55cda56ce625" containerName="revoke-keys" Oct 30 21:42:20.467471 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.467447 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019e5ae-x55jc" Oct 30 21:42:20.475060 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.475028 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a019e5ae-x55jc"] Oct 30 21:42:20.529830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.495233 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4600e16-207d-45f5-bbb4-55cda56ce625-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4600e16-207d-45f5-bbb4-55cda56ce625-maas-api-xks-ca\") pod \"c4600e16-207d-45f5-bbb4-55cda56ce625\" (UID: \"c4600e16-207d-45f5-bbb4-55cda56ce625\") " Oct 30 21:42:20.529830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.495391 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4600e16-207d-45f5-bbb4-55cda56ce625-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4600e16-207d-45f5-bbb4-55cda56ce625-maas-api-openshift-ca\") pod \"c4600e16-207d-45f5-bbb4-55cda56ce625\" (UID: \"c4600e16-207d-45f5-bbb4-55cda56ce625\") " Oct 30 21:42:20.529830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.495507 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4600e16-207d-45f5-bbb4-55cda56ce625-kube-api-access-wq2px\" (UniqueName: \"kubernetes.io/projected/c4600e16-207d-45f5-bbb4-55cda56ce625-kube-api-access-wq2px\") pod \"c4600e16-207d-45f5-bbb4-55cda56ce625\" (UID: \"c4600e16-207d-45f5-bbb4-55cda56ce625\") " Oct 30 21:42:20.529830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.499740 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4600e16-207d-45f5-bbb4-55cda56ce625-maas-api-openshift-ca" pod "c4600e16-207d-45f5-bbb4-55cda56ce625" (UID: "c4600e16-207d-45f5-bbb4-55cda56ce625"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:20.529830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.526447 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4600e16-207d-45f5-bbb4-55cda56ce625-kube-api-access-wq2px" pod "c4600e16-207d-45f5-bbb4-55cda56ce625" (UID: "c4600e16-207d-45f5-bbb4-55cda56ce625"). InnerVolumeSpecName "kube-api-access-wq2px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:20.554810 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.554598 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4600e16-207d-45f5-bbb4-55cda56ce625-maas-api-xks-ca" pod "c4600e16-207d-45f5-bbb4-55cda56ce625" (UID: "c4600e16-207d-45f5-bbb4-55cda56ce625"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:42:20.598026 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.597959 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f7a9d0-422c-45ad-a928-82a851cbbd34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a019e5ae-x55jc\" (UID: \"26f7a9d0-422c-45ad-a928-82a851cbbd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019e5ae-x55jc" Oct 30 21:42:20.598282 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.598262 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6mq4\" (UniqueName: \"kubernetes.io/projected/26f7a9d0-422c-45ad-a928-82a851cbbd34-kube-api-access-s6mq4\") pod \"maas-api-revoke-sub-a019e5ae-x55jc\" (UID: \"26f7a9d0-422c-45ad-a928-82a851cbbd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019e5ae-x55jc" Oct 30 21:42:20.598360 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.598345 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f7a9d0-422c-45ad-a928-82a851cbbd34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a019e5ae-x55jc\" (UID: \"26f7a9d0-422c-45ad-a928-82a851cbbd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019e5ae-x55jc" Oct 30 21:42:20.598642 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.598619 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq2px\" (UniqueName: \"kubernetes.io/projected/c4600e16-207d-45f5-bbb4-55cda56ce625-kube-api-access-wq2px\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:20.598720 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.598648 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4600e16-207d-45f5-bbb4-55cda56ce625-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:20.598720 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.598665 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4600e16-207d-45f5-bbb4-55cda56ce625-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:20.603565 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:42:20.603535 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545f6262_2b94_4b51_9688_d08feb89cb59.slice/crio-f8a985f77c2cc4162059822e81ccaa5c624005f5764001e417f81efc2cd09d78 WatchSource:0}: Error finding container f8a985f77c2cc4162059822e81ccaa5c624005f5764001e417f81efc2cd09d78: Status 404 returned error can't find the container with id f8a985f77c2cc4162059822e81ccaa5c624005f5764001e417f81efc2cd09d78 Oct 30 21:42:20.605451 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.605427 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd3352f7-vkjzp"] Oct 30 21:42:20.700267 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.700229 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f7a9d0-422c-45ad-a928-82a851cbbd34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a019e5ae-x55jc\" (UID: \"26f7a9d0-422c-45ad-a928-82a851cbbd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019e5ae-x55jc" Oct 30 21:42:20.700506 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.700364 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6mq4\" (UniqueName: \"kubernetes.io/projected/26f7a9d0-422c-45ad-a928-82a851cbbd34-kube-api-access-s6mq4\") pod \"maas-api-revoke-sub-a019e5ae-x55jc\" (UID: \"26f7a9d0-422c-45ad-a928-82a851cbbd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019e5ae-x55jc" Oct 30 21:42:20.700506 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.700418 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f7a9d0-422c-45ad-a928-82a851cbbd34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a019e5ae-x55jc\" (UID: \"26f7a9d0-422c-45ad-a928-82a851cbbd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019e5ae-x55jc" Oct 30 21:42:20.701317 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.701292 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f7a9d0-422c-45ad-a928-82a851cbbd34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a019e5ae-x55jc\" (UID: \"26f7a9d0-422c-45ad-a928-82a851cbbd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019e5ae-x55jc" Oct 30 21:42:20.703100 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.703081 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f7a9d0-422c-45ad-a928-82a851cbbd34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a019e5ae-x55jc\" (UID: \"26f7a9d0-422c-45ad-a928-82a851cbbd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019e5ae-x55jc" Oct 30 21:42:20.712920 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.712859 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6mq4\" (UniqueName: \"kubernetes.io/projected/26f7a9d0-422c-45ad-a928-82a851cbbd34-kube-api-access-s6mq4\") pod \"maas-api-revoke-sub-a019e5ae-x55jc\" (UID: \"26f7a9d0-422c-45ad-a928-82a851cbbd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019e5ae-x55jc" Oct 30 21:42:20.816481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:20.816446 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019e5ae-x55jc" Oct 30 21:42:21.062883 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:21.062805 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7351-51e6cb"] Oct 30 21:42:21.070744 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:21.070711 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7351-51e6cb"] Oct 30 21:42:21.112934 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:21.112892 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a019e5ae-x55jc"] Oct 30 21:42:21.218032 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:21.218007 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9aa0-4rk86" Oct 30 21:42:21.218032 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:21.218016 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9aa0-4rk86" event={"ID":"c4600e16-207d-45f5-bbb4-55cda56ce625","Type":"ContainerDied","Data":"d9546c9b90204ca29d843666c36127e30adca46c69a58f5ad0fe3475a79c6596"} Oct 30 21:42:21.218574 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:21.218072 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9546c9b90204ca29d843666c36127e30adca46c69a58f5ad0fe3475a79c6596" Oct 30 21:42:21.219524 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:21.219492 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019e5ae-x55jc" event={"ID":"26f7a9d0-422c-45ad-a928-82a851cbbd34","Type":"ContainerStarted","Data":"b481fb80bf852fea199e16d5629095431fb79710e17a2daac0ee1be1dcb15437"} Oct 30 21:42:21.221327 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:21.221295 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3352f7-vkjzp" event={"ID":"545f6262-2b94-4b51-9688-d08feb89cb59","Type":"ContainerStarted","Data":"b813992fa60129a14f2e0020788330e386868677809b4153e671d434a1b7e9bb"} Oct 30 21:42:21.221458 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:21.221337 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3352f7-vkjzp" event={"ID":"545f6262-2b94-4b51-9688-d08feb89cb59","Type":"ContainerStarted","Data":"f8a985f77c2cc4162059822e81ccaa5c624005f5764001e417f81efc2cd09d78"} Oct 30 21:42:21.243500 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:21.243439 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3352f7-vkjzp" podStartSLOduration=2.24342592 podStartE2EDuration="2.24342592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:21.238907006 +0000 UTC m=+809.834723666" watchObservedRunningTime="2026-10-30 21:42:21.24342592 +0000 UTC m=+809.839242581" Oct 30 21:42:21.339212 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:21.339133 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:22.050068 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:22.050034 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:22.235508 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:22.235465 5801 generic.go:356] "Generic (PLEG): container finished" podID="26f7a9d0-422c-45ad-a928-82a851cbbd34" containerID="e441d15a996b438ec80296ecf826dd0ece631e687e45c22b53048ca6bcea0e55" exitCode=0 Oct 30 21:42:22.236059 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:22.235554 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019e5ae-x55jc" event={"ID":"26f7a9d0-422c-45ad-a928-82a851cbbd34","Type":"ContainerDied","Data":"e441d15a996b438ec80296ecf826dd0ece631e687e45c22b53048ca6bcea0e55"} Oct 30 21:42:22.262627 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:22.262411 5801 generic.go:356] "Generic (PLEG): container finished" podID="545f6262-2b94-4b51-9688-d08feb89cb59" containerID="b813992fa60129a14f2e0020788330e386868677809b4153e671d434a1b7e9bb" exitCode=0 Oct 30 21:42:22.262627 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:22.262556 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3352f7-vkjzp" event={"ID":"545f6262-2b94-4b51-9688-d08feb89cb59","Type":"ContainerDied","Data":"b813992fa60129a14f2e0020788330e386868677809b4153e671d434a1b7e9bb"} Oct 30 21:42:22.262904 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:22.262877 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7351-51e6cb" podUID="486c5963-3aad-47de-b5de-23595e3abec1" containerName="test-curl-7351-51e6cb" containerID="cri-o://b3caa783cab09026bbe8f6fa88438eaaaac9a80b22768ed71cada604068ed99e" gracePeriod=2 Oct 30 21:42:22.660380 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:22.660330 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:22.771226 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:22.771186 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:22.872311 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:22.872263 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:22.967722 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:22.967655 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa8e3a54-z56h7"] Oct 30 21:42:22.968581 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:22.968562 5801 cpu_manager.go:414] "containerMap: removing container" podUID="486c5963-3aad-47de-b5de-23595e3abec1" containerName="test-curl-7351-51e6cb" Oct 30 21:42:22.968631 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:22.968584 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="486c5963-3aad-47de-b5de-23595e3abec1" containerName="test-curl-7351-51e6cb" Oct 30 21:42:22.968714 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:22.968705 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="486c5963-3aad-47de-b5de-23595e3abec1" containerName="test-curl-7351-51e6cb" Oct 30 21:42:22.972708 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:22.972691 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8e3a54-z56h7" Oct 30 21:42:22.982567 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:22.982537 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa8e3a54-z56h7"] Oct 30 21:42:23.049981 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.049709 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2efd9b31-0b1e-4b5a-8abd-2041520bc18a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa8e3a54-z56h7\" (UID: \"2efd9b31-0b1e-4b5a-8abd-2041520bc18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8e3a54-z56h7" Oct 30 21:42:23.080320 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.050873 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2efd9b31-0b1e-4b5a-8abd-2041520bc18a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa8e3a54-z56h7\" (UID: \"2efd9b31-0b1e-4b5a-8abd-2041520bc18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8e3a54-z56h7" Oct 30 21:42:23.080320 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.051022 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rw5r\" (UniqueName: \"kubernetes.io/projected/2efd9b31-0b1e-4b5a-8abd-2041520bc18a-kube-api-access-2rw5r\") pod \"maas-api-revoke-sub-fa8e3a54-z56h7\" (UID: \"2efd9b31-0b1e-4b5a-8abd-2041520bc18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8e3a54-z56h7" Oct 30 21:42:23.153203 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.153175 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2efd9b31-0b1e-4b5a-8abd-2041520bc18a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa8e3a54-z56h7\" (UID: \"2efd9b31-0b1e-4b5a-8abd-2041520bc18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8e3a54-z56h7" Oct 30 21:42:23.153362 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.153269 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rw5r\" (UniqueName: \"kubernetes.io/projected/2efd9b31-0b1e-4b5a-8abd-2041520bc18a-kube-api-access-2rw5r\") pod \"maas-api-revoke-sub-fa8e3a54-z56h7\" (UID: \"2efd9b31-0b1e-4b5a-8abd-2041520bc18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8e3a54-z56h7" Oct 30 21:42:23.153498 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.153479 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2efd9b31-0b1e-4b5a-8abd-2041520bc18a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa8e3a54-z56h7\" (UID: \"2efd9b31-0b1e-4b5a-8abd-2041520bc18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8e3a54-z56h7" Oct 30 21:42:23.153992 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.153968 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2efd9b31-0b1e-4b5a-8abd-2041520bc18a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa8e3a54-z56h7\" (UID: \"2efd9b31-0b1e-4b5a-8abd-2041520bc18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8e3a54-z56h7" Oct 30 21:42:23.155887 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.155860 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2efd9b31-0b1e-4b5a-8abd-2041520bc18a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa8e3a54-z56h7\" (UID: \"2efd9b31-0b1e-4b5a-8abd-2041520bc18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8e3a54-z56h7" Oct 30 21:42:23.166478 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.166455 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rw5r\" (UniqueName: \"kubernetes.io/projected/2efd9b31-0b1e-4b5a-8abd-2041520bc18a-kube-api-access-2rw5r\") pod \"maas-api-revoke-sub-fa8e3a54-z56h7\" (UID: \"2efd9b31-0b1e-4b5a-8abd-2041520bc18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8e3a54-z56h7" Oct 30 21:42:23.293493 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.293461 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8e3a54-z56h7" Oct 30 21:42:23.325309 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.325210 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:23.461784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.456717 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3352f7-vkjzp" Oct 30 21:42:23.491805 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.491781 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019e5ae-x55jc" Oct 30 21:42:23.568257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.568227 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26f7a9d0-422c-45ad-a928-82a851cbbd34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f7a9d0-422c-45ad-a928-82a851cbbd34-maas-api-openshift-ca\") pod \"26f7a9d0-422c-45ad-a928-82a851cbbd34\" (UID: \"26f7a9d0-422c-45ad-a928-82a851cbbd34\") " Oct 30 21:42:23.568431 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.568321 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/545f6262-2b94-4b51-9688-d08feb89cb59-kube-api-access-fwfst\" (UniqueName: \"kubernetes.io/projected/545f6262-2b94-4b51-9688-d08feb89cb59-kube-api-access-fwfst\") pod \"545f6262-2b94-4b51-9688-d08feb89cb59\" (UID: \"545f6262-2b94-4b51-9688-d08feb89cb59\") " Oct 30 21:42:23.568431 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.568381 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/545f6262-2b94-4b51-9688-d08feb89cb59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545f6262-2b94-4b51-9688-d08feb89cb59-maas-api-openshift-ca\") pod \"545f6262-2b94-4b51-9688-d08feb89cb59\" (UID: \"545f6262-2b94-4b51-9688-d08feb89cb59\") " Oct 30 21:42:23.568549 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.568492 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f7a9d0-422c-45ad-a928-82a851cbbd34-kube-api-access-s6mq4\" (UniqueName: \"kubernetes.io/projected/26f7a9d0-422c-45ad-a928-82a851cbbd34-kube-api-access-s6mq4\") pod \"26f7a9d0-422c-45ad-a928-82a851cbbd34\" (UID: \"26f7a9d0-422c-45ad-a928-82a851cbbd34\") " Oct 30 21:42:23.568603 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.568578 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26f7a9d0-422c-45ad-a928-82a851cbbd34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f7a9d0-422c-45ad-a928-82a851cbbd34-maas-api-xks-ca\") pod \"26f7a9d0-422c-45ad-a928-82a851cbbd34\" (UID: \"26f7a9d0-422c-45ad-a928-82a851cbbd34\") " Oct 30 21:42:23.568658 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.568590 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f7a9d0-422c-45ad-a928-82a851cbbd34-maas-api-openshift-ca" pod "26f7a9d0-422c-45ad-a928-82a851cbbd34" (UID: "26f7a9d0-422c-45ad-a928-82a851cbbd34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:23.568658 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.568624 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/545f6262-2b94-4b51-9688-d08feb89cb59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545f6262-2b94-4b51-9688-d08feb89cb59-maas-api-xks-ca\") pod \"545f6262-2b94-4b51-9688-d08feb89cb59\" (UID: \"545f6262-2b94-4b51-9688-d08feb89cb59\") " Oct 30 21:42:23.568797 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.568661 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/545f6262-2b94-4b51-9688-d08feb89cb59-maas-api-openshift-ca" pod "545f6262-2b94-4b51-9688-d08feb89cb59" (UID: "545f6262-2b94-4b51-9688-d08feb89cb59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:23.569632 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.569596 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f7a9d0-422c-45ad-a928-82a851cbbd34-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:23.569632 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.569628 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545f6262-2b94-4b51-9688-d08feb89cb59-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:23.570631 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.570608 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f7a9d0-422c-45ad-a928-82a851cbbd34-kube-api-access-s6mq4" pod "26f7a9d0-422c-45ad-a928-82a851cbbd34" (UID: "26f7a9d0-422c-45ad-a928-82a851cbbd34"). InnerVolumeSpecName "kube-api-access-s6mq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:23.570631 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.570623 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/545f6262-2b94-4b51-9688-d08feb89cb59-maas-api-xks-ca" pod "545f6262-2b94-4b51-9688-d08feb89cb59" (UID: "545f6262-2b94-4b51-9688-d08feb89cb59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:42:23.570818 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.570674 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545f6262-2b94-4b51-9688-d08feb89cb59-kube-api-access-fwfst" pod "545f6262-2b94-4b51-9688-d08feb89cb59" (UID: "545f6262-2b94-4b51-9688-d08feb89cb59"). InnerVolumeSpecName "kube-api-access-fwfst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:23.570818 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.570675 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26f7a9d0-422c-45ad-a928-82a851cbbd34-maas-api-xks-ca" pod "26f7a9d0-422c-45ad-a928-82a851cbbd34" (UID: "26f7a9d0-422c-45ad-a928-82a851cbbd34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:42:23.622169 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:42:23.622142 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2efd9b31_0b1e_4b5a_8abd_2041520bc18a.slice/crio-85bf286131532a0dc3ba4908dbe0949f9405451734851c07727d8fe8eb4f52e3 WatchSource:0}: Error finding container 85bf286131532a0dc3ba4908dbe0949f9405451734851c07727d8fe8eb4f52e3: Status 404 returned error can't find the container with id 85bf286131532a0dc3ba4908dbe0949f9405451734851c07727d8fe8eb4f52e3 Oct 30 21:42:23.623607 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.623583 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa8e3a54-z56h7"] Oct 30 21:42:23.670967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.670945 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwfst\" (UniqueName: \"kubernetes.io/projected/545f6262-2b94-4b51-9688-d08feb89cb59-kube-api-access-fwfst\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:23.670967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.670968 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6mq4\" (UniqueName: \"kubernetes.io/projected/26f7a9d0-422c-45ad-a928-82a851cbbd34-kube-api-access-s6mq4\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:23.671089 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.670977 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f7a9d0-422c-45ad-a928-82a851cbbd34-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:23.671089 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.670987 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545f6262-2b94-4b51-9688-d08feb89cb59-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:23.736410 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:23.736365 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:24.286551 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:24.286522 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8e3a54-z56h7" event={"ID":"2efd9b31-0b1e-4b5a-8abd-2041520bc18a","Type":"ContainerStarted","Data":"bd409099995303af1e2c10d8d466ef003af51aba2043407286b44875dbcadae8"} Oct 30 21:42:24.286708 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:24.286578 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8e3a54-z56h7" event={"ID":"2efd9b31-0b1e-4b5a-8abd-2041520bc18a","Type":"ContainerStarted","Data":"85bf286131532a0dc3ba4908dbe0949f9405451734851c07727d8fe8eb4f52e3"} Oct 30 21:42:24.288276 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:24.288255 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019e5ae-x55jc" Oct 30 21:42:24.288276 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:24.288265 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019e5ae-x55jc" event={"ID":"26f7a9d0-422c-45ad-a928-82a851cbbd34","Type":"ContainerDied","Data":"b481fb80bf852fea199e16d5629095431fb79710e17a2daac0ee1be1dcb15437"} Oct 30 21:42:24.288442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:24.288301 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b481fb80bf852fea199e16d5629095431fb79710e17a2daac0ee1be1dcb15437" Oct 30 21:42:24.289927 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:24.289892 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3352f7-vkjzp" event={"ID":"545f6262-2b94-4b51-9688-d08feb89cb59","Type":"ContainerDied","Data":"f8a985f77c2cc4162059822e81ccaa5c624005f5764001e417f81efc2cd09d78"} Oct 30 21:42:24.290036 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:24.289940 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8a985f77c2cc4162059822e81ccaa5c624005f5764001e417f81efc2cd09d78" Oct 30 21:42:24.290036 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:24.289901 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3352f7-vkjzp" Oct 30 21:42:24.306724 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:24.306682 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8e3a54-z56h7" podStartSLOduration=2.306671177 podStartE2EDuration="2.306671177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:24.304648577 +0000 UTC m=+812.900465264" watchObservedRunningTime="2026-10-30 21:42:24.306671177 +0000 UTC m=+812.902487845" Oct 30 21:42:24.456618 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:24.456586 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:24.477938 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:24.477913 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7351-51e6cb" Oct 30 21:42:24.589052 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:24.589003 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486c5963-3aad-47de-b5de-23595e3abec1-kube-api-access-cz9qz\" (UniqueName: \"kubernetes.io/projected/486c5963-3aad-47de-b5de-23595e3abec1-kube-api-access-cz9qz\") pod \"486c5963-3aad-47de-b5de-23595e3abec1\" (UID: \"486c5963-3aad-47de-b5de-23595e3abec1\") " Oct 30 21:42:24.591480 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:24.591452 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486c5963-3aad-47de-b5de-23595e3abec1-kube-api-access-cz9qz" pod "486c5963-3aad-47de-b5de-23595e3abec1" (UID: "486c5963-3aad-47de-b5de-23595e3abec1"). InnerVolumeSpecName "kube-api-access-cz9qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:24.691233 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:24.691206 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz9qz\" (UniqueName: \"kubernetes.io/projected/486c5963-3aad-47de-b5de-23595e3abec1-kube-api-access-cz9qz\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:25.205209 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:25.204671 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:25.304966 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:25.304930 5801 generic.go:356] "Generic (PLEG): container finished" podID="486c5963-3aad-47de-b5de-23595e3abec1" containerID="b3caa783cab09026bbe8f6fa88438eaaaac9a80b22768ed71cada604068ed99e" exitCode=137 Oct 30 21:42:25.305162 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:25.304996 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7351-51e6cb" Oct 30 21:42:25.305162 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:25.305024 5801 scope.go:122] "RemoveContainer" containerID="b3caa783cab09026bbe8f6fa88438eaaaac9a80b22768ed71cada604068ed99e" Oct 30 21:42:25.307114 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:25.307092 5801 generic.go:356] "Generic (PLEG): container finished" podID="2efd9b31-0b1e-4b5a-8abd-2041520bc18a" containerID="bd409099995303af1e2c10d8d466ef003af51aba2043407286b44875dbcadae8" exitCode=0 Oct 30 21:42:25.307559 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:25.307186 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8e3a54-z56h7" event={"ID":"2efd9b31-0b1e-4b5a-8abd-2041520bc18a","Type":"ContainerDied","Data":"bd409099995303af1e2c10d8d466ef003af51aba2043407286b44875dbcadae8"} Oct 30 21:42:25.317850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:25.317668 5801 scope.go:122] "RemoveContainer" containerID="b3caa783cab09026bbe8f6fa88438eaaaac9a80b22768ed71cada604068ed99e" Oct 30 21:42:25.318035 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:42:25.318007 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3caa783cab09026bbe8f6fa88438eaaaac9a80b22768ed71cada604068ed99e\": container with ID starting with b3caa783cab09026bbe8f6fa88438eaaaac9a80b22768ed71cada604068ed99e not found: ID does not exist" containerID="b3caa783cab09026bbe8f6fa88438eaaaac9a80b22768ed71cada604068ed99e" Oct 30 21:42:25.318117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:25.318047 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3caa783cab09026bbe8f6fa88438eaaaac9a80b22768ed71cada604068ed99e"} err="failed to get container status \"b3caa783cab09026bbe8f6fa88438eaaaac9a80b22768ed71cada604068ed99e\": rpc error: code = NotFound desc = could not find container \"b3caa783cab09026bbe8f6fa88438eaaaac9a80b22768ed71cada604068ed99e\": container with ID starting with b3caa783cab09026bbe8f6fa88438eaaaac9a80b22768ed71cada604068ed99e not found: ID does not exist" Oct 30 21:42:25.335762 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:42:25.335722 5801 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7351-51e6cb\" is forbidden: User \"system:node:ip-10-0-140-163.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-163.us-east-2.compute.internal' and this object" podUID="486c5963-3aad-47de-b5de-23595e3abec1" pod="opendatahub/test-curl-7351-51e6cb" Oct 30 21:42:25.525661 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:25.525573 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="486c5963-3aad-47de-b5de-23595e3abec1" path="/var/lib/kubelet/pods/486c5963-3aad-47de-b5de-23595e3abec1/volumes" Oct 30 21:42:25.849447 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:25.849409 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:26.489227 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:26.489201 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8e3a54-z56h7" Oct 30 21:42:26.546865 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:26.546825 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:26.635122 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:26.635031 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2efd9b31-0b1e-4b5a-8abd-2041520bc18a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2efd9b31-0b1e-4b5a-8abd-2041520bc18a-maas-api-openshift-ca\") pod \"2efd9b31-0b1e-4b5a-8abd-2041520bc18a\" (UID: \"2efd9b31-0b1e-4b5a-8abd-2041520bc18a\") " Oct 30 21:42:26.635122 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:26.635098 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2efd9b31-0b1e-4b5a-8abd-2041520bc18a-kube-api-access-2rw5r\" (UniqueName: \"kubernetes.io/projected/2efd9b31-0b1e-4b5a-8abd-2041520bc18a-kube-api-access-2rw5r\") pod \"2efd9b31-0b1e-4b5a-8abd-2041520bc18a\" (UID: \"2efd9b31-0b1e-4b5a-8abd-2041520bc18a\") " Oct 30 21:42:26.635453 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:26.635300 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2efd9b31-0b1e-4b5a-8abd-2041520bc18a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2efd9b31-0b1e-4b5a-8abd-2041520bc18a-maas-api-xks-ca\") pod \"2efd9b31-0b1e-4b5a-8abd-2041520bc18a\" (UID: \"2efd9b31-0b1e-4b5a-8abd-2041520bc18a\") " Oct 30 21:42:26.635521 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:26.635460 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2efd9b31-0b1e-4b5a-8abd-2041520bc18a-maas-api-openshift-ca" pod "2efd9b31-0b1e-4b5a-8abd-2041520bc18a" (UID: "2efd9b31-0b1e-4b5a-8abd-2041520bc18a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:26.636507 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:26.636315 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2efd9b31-0b1e-4b5a-8abd-2041520bc18a-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:26.637877 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:26.637849 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2efd9b31-0b1e-4b5a-8abd-2041520bc18a-maas-api-xks-ca" pod "2efd9b31-0b1e-4b5a-8abd-2041520bc18a" (UID: "2efd9b31-0b1e-4b5a-8abd-2041520bc18a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:42:26.638066 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:26.638048 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2efd9b31-0b1e-4b5a-8abd-2041520bc18a-kube-api-access-2rw5r" pod "2efd9b31-0b1e-4b5a-8abd-2041520bc18a" (UID: "2efd9b31-0b1e-4b5a-8abd-2041520bc18a"). InnerVolumeSpecName "kube-api-access-2rw5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:26.737603 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:26.737569 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2efd9b31-0b1e-4b5a-8abd-2041520bc18a-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:26.737603 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:26.737602 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rw5r\" (UniqueName: \"kubernetes.io/projected/2efd9b31-0b1e-4b5a-8abd-2041520bc18a-kube-api-access-2rw5r\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:26.751604 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:26.751547 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:27.282031 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:27.281991 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:27.364545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:27.364498 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8e3a54-z56h7" event={"ID":"2efd9b31-0b1e-4b5a-8abd-2041520bc18a","Type":"ContainerDied","Data":"85bf286131532a0dc3ba4908dbe0949f9405451734851c07727d8fe8eb4f52e3"} Oct 30 21:42:27.364856 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:27.364835 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85bf286131532a0dc3ba4908dbe0949f9405451734851c07727d8fe8eb4f52e3" Oct 30 21:42:27.365051 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:27.365037 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8e3a54-z56h7" Oct 30 21:42:27.826357 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:27.826324 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:27.934162 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:27.934123 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:28.526333 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:28.526299 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:29.223382 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:29.223337 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:29.897629 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:29.897590 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:30.002361 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.002321 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:30.126883 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.126853 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:30.295272 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:42:30.295129 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:42:30.320248 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.320211 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:30.435309 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.435268 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:30.581176 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.581149 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:30.705106 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.705075 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfcbbd3b-rh46j"] Oct 30 21:42:30.705761 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.705730 5801 cpu_manager.go:414] "containerMap: removing container" podUID="2efd9b31-0b1e-4b5a-8abd-2041520bc18a" containerName="revoke-keys" Oct 30 21:42:30.705761 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.705747 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2efd9b31-0b1e-4b5a-8abd-2041520bc18a" containerName="revoke-keys" Oct 30 21:42:30.705925 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.705778 5801 cpu_manager.go:414] "containerMap: removing container" podUID="26f7a9d0-422c-45ad-a928-82a851cbbd34" containerName="revoke-keys" Oct 30 21:42:30.705925 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.705784 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f7a9d0-422c-45ad-a928-82a851cbbd34" containerName="revoke-keys" Oct 30 21:42:30.705925 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.705843 5801 cpu_manager.go:414] "containerMap: removing container" podUID="545f6262-2b94-4b51-9688-d08feb89cb59" containerName="revoke-keys" Oct 30 21:42:30.705925 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.705851 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="545f6262-2b94-4b51-9688-d08feb89cb59" containerName="revoke-keys" Oct 30 21:42:30.706065 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.705971 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="545f6262-2b94-4b51-9688-d08feb89cb59" containerName="revoke-keys" Oct 30 21:42:30.706065 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.705983 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="2efd9b31-0b1e-4b5a-8abd-2041520bc18a" containerName="revoke-keys" Oct 30 21:42:30.706065 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.705994 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f7a9d0-422c-45ad-a928-82a851cbbd34" containerName="revoke-keys" Oct 30 21:42:30.712542 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.712521 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcbbd3b-rh46j" Oct 30 21:42:30.724348 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.724322 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:42:30.724504 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.724484 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:42:30.724965 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.724785 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:42:30.729666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.729639 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfcbbd3b-rh46j"] Oct 30 21:42:30.732848 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.731996 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:42:30.751348 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.751325 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:30.819886 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.819861 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t266q\" (UniqueName: \"kubernetes.io/projected/0015a27a-eba7-4ed4-8159-4dad866c5211-kube-api-access-t266q\") pod \"maas-api-revoke-sub-cfcbbd3b-rh46j\" (UID: \"0015a27a-eba7-4ed4-8159-4dad866c5211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcbbd3b-rh46j" Oct 30 21:42:30.820040 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.820021 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0015a27a-eba7-4ed4-8159-4dad866c5211-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfcbbd3b-rh46j\" (UID: \"0015a27a-eba7-4ed4-8159-4dad866c5211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcbbd3b-rh46j" Oct 30 21:42:30.820086 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.820071 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0015a27a-eba7-4ed4-8159-4dad866c5211-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfcbbd3b-rh46j\" (UID: \"0015a27a-eba7-4ed4-8159-4dad866c5211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcbbd3b-rh46j" Oct 30 21:42:30.883513 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.883428 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:30.921873 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.921844 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0015a27a-eba7-4ed4-8159-4dad866c5211-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfcbbd3b-rh46j\" (UID: \"0015a27a-eba7-4ed4-8159-4dad866c5211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcbbd3b-rh46j" Oct 30 21:42:30.922073 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.922056 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t266q\" (UniqueName: \"kubernetes.io/projected/0015a27a-eba7-4ed4-8159-4dad866c5211-kube-api-access-t266q\") pod \"maas-api-revoke-sub-cfcbbd3b-rh46j\" (UID: \"0015a27a-eba7-4ed4-8159-4dad866c5211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcbbd3b-rh46j" Oct 30 21:42:30.922221 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.922208 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0015a27a-eba7-4ed4-8159-4dad866c5211-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfcbbd3b-rh46j\" (UID: \"0015a27a-eba7-4ed4-8159-4dad866c5211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcbbd3b-rh46j" Oct 30 21:42:30.922534 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.922514 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0015a27a-eba7-4ed4-8159-4dad866c5211-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfcbbd3b-rh46j\" (UID: \"0015a27a-eba7-4ed4-8159-4dad866c5211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcbbd3b-rh46j" Oct 30 21:42:30.924489 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.924464 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0015a27a-eba7-4ed4-8159-4dad866c5211-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfcbbd3b-rh46j\" (UID: \"0015a27a-eba7-4ed4-8159-4dad866c5211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcbbd3b-rh46j" Oct 30 21:42:30.948077 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:30.948050 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t266q\" (UniqueName: \"kubernetes.io/projected/0015a27a-eba7-4ed4-8159-4dad866c5211-kube-api-access-t266q\") pod \"maas-api-revoke-sub-cfcbbd3b-rh46j\" (UID: \"0015a27a-eba7-4ed4-8159-4dad866c5211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcbbd3b-rh46j" Oct 30 21:42:31.024664 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.024627 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79"] Oct 30 21:42:31.030888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.030863 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.031008 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.030977 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcbbd3b-rh46j" Oct 30 21:42:31.042312 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.042083 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-938e67d1-gw-openshift-default-dockercfg-xbdxx\"" Oct 30 21:42:31.048722 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.048700 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79"] Oct 30 21:42:31.084479 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.084452 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:31.127767 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.127720 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-credential-socket\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.127897 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.127809 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts5vh\" (UniqueName: \"kubernetes.io/projected/c2c309a8-0866-464c-be99-67d454090ea6-kube-api-access-ts5vh\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.127963 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.127911 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-istio-data\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.128020 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.127975 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-workload-socket\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.128317 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.128242 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c309a8-0866-464c-be99-67d454090ea6-istiod-ca-cert\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.128317 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.128299 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c309a8-0866-464c-be99-67d454090ea6-istio-podinfo\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.128465 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.128346 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c309a8-0866-464c-be99-67d454090ea6-istio-token\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.128465 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.128388 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-istio-envoy\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.128465 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.128421 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-workload-certs\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.272827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.266072 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-credential-socket\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.272827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.266151 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts5vh\" (UniqueName: \"kubernetes.io/projected/c2c309a8-0866-464c-be99-67d454090ea6-kube-api-access-ts5vh\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.272827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.266277 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-istio-data\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.272827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.266347 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-workload-socket\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.272827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.266538 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c309a8-0866-464c-be99-67d454090ea6-istiod-ca-cert\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.272827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.266585 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c309a8-0866-464c-be99-67d454090ea6-istio-podinfo\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.272827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.266635 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c309a8-0866-464c-be99-67d454090ea6-istio-token\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.272827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.266674 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-istio-envoy\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.272827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.266710 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-workload-certs\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.272827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.267373 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-workload-certs\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.272827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.267644 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-credential-socket\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.272827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.268244 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-istio-data\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.272827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.268479 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-workload-socket\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.272827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.269145 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c309a8-0866-464c-be99-67d454090ea6-istiod-ca-cert\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.308273 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.306389 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c309a8-0866-464c-be99-67d454090ea6-istio-podinfo\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.457386 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.457339 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfcbbd3b-rh46j"] Oct 30 21:42:31.461511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.461455 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-istio-envoy\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.468850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.462084 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c309a8-0866-464c-be99-67d454090ea6-istio-token\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.485429 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.485403 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts5vh\" (UniqueName: \"kubernetes.io/projected/c2c309a8-0866-464c-be99-67d454090ea6-kube-api-access-ts5vh\") pod \"e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:31.515014 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.514988 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:31.610674 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.610638 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:31.693208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:31.693169 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:32.012897 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:42:32.012852 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c309a8_0866_464c_be99_67d454090ea6.slice/crio-adcf86b79f183c6ebdea09c8ced257baedb375074cc64baa1c331e45274461cb WatchSource:0}: Error finding container adcf86b79f183c6ebdea09c8ced257baedb375074cc64baa1c331e45274461cb: Status 404 returned error can't find the container with id adcf86b79f183c6ebdea09c8ced257baedb375074cc64baa1c331e45274461cb Oct 30 21:42:32.013372 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.013339 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79"] Oct 30 21:42:32.015368 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.015316 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:42:32.015558 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.015537 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:42:32.015697 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.015674 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:42:32.023958 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.020310 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:32.083472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.083447 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s"] Oct 30 21:42:32.089651 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.089627 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s" Oct 30 21:42:32.099436 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.099409 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s"] Oct 30 21:42:32.105967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.103643 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e19227f8\"" Oct 30 21:42:32.136542 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.136468 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:32.141173 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.140524 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn"] Oct 30 21:42:32.146887 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.146866 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn" Oct 30 21:42:32.161048 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.161025 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e19227f8-dockercfg-b7cf5\"" Oct 30 21:42:32.173231 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.173209 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn"] Oct 30 21:42:32.202817 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.202782 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqmpc\" (UniqueName: \"kubernetes.io/projected/6f0d6f19-bad5-415d-8230-125bf39ae64f-kube-api-access-jqmpc\") pod \"payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s\" (UID: \"6f0d6f19-bad5-415d-8230-125bf39ae64f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s" Oct 30 21:42:32.203014 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.202941 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0d6f19-bad5-415d-8230-125bf39ae64f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s\" (UID: \"6f0d6f19-bad5-415d-8230-125bf39ae64f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s" Oct 30 21:42:32.393617 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.329545 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cklx9\" (UniqueName: \"kubernetes.io/projected/3b377cbc-6859-4ae1-bde2-bac7680bb7d2-kube-api-access-cklx9\") pod \"payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn\" (UID: \"3b377cbc-6859-4ae1-bde2-bac7680bb7d2\") " pod="openshift-ingress/payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn" Oct 30 21:42:32.393617 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.329891 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmpc\" (UniqueName: \"kubernetes.io/projected/6f0d6f19-bad5-415d-8230-125bf39ae64f-kube-api-access-jqmpc\") pod \"payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s\" (UID: \"6f0d6f19-bad5-415d-8230-125bf39ae64f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s" Oct 30 21:42:32.393617 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.329994 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0d6f19-bad5-415d-8230-125bf39ae64f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s\" (UID: \"6f0d6f19-bad5-415d-8230-125bf39ae64f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s" Oct 30 21:42:32.393617 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.330038 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b377cbc-6859-4ae1-bde2-bac7680bb7d2-plugins-config-volume\") pod \"payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn\" (UID: \"3b377cbc-6859-4ae1-bde2-bac7680bb7d2\") " pod="openshift-ingress/payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn" Oct 30 21:42:32.393617 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.331371 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0d6f19-bad5-415d-8230-125bf39ae64f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s\" (UID: \"6f0d6f19-bad5-415d-8230-125bf39ae64f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s" Oct 30 21:42:32.432567 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.432529 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqmpc\" (UniqueName: \"kubernetes.io/projected/6f0d6f19-bad5-415d-8230-125bf39ae64f-kube-api-access-jqmpc\") pod \"payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s\" (UID: \"6f0d6f19-bad5-415d-8230-125bf39ae64f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s" Oct 30 21:42:32.444745 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.444703 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b377cbc-6859-4ae1-bde2-bac7680bb7d2-plugins-config-volume\") pod \"payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn\" (UID: \"3b377cbc-6859-4ae1-bde2-bac7680bb7d2\") " pod="openshift-ingress/payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn" Oct 30 21:42:32.449204 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.449137 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cklx9\" (UniqueName: \"kubernetes.io/projected/3b377cbc-6859-4ae1-bde2-bac7680bb7d2-kube-api-access-cklx9\") pod \"payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn\" (UID: \"3b377cbc-6859-4ae1-bde2-bac7680bb7d2\") " pod="openshift-ingress/payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn" Oct 30 21:42:32.450384 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.450348 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b377cbc-6859-4ae1-bde2-bac7680bb7d2-plugins-config-volume\") pod \"payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn\" (UID: \"3b377cbc-6859-4ae1-bde2-bac7680bb7d2\") " pod="openshift-ingress/payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn" Oct 30 21:42:32.508024 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.507673 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cklx9\" (UniqueName: \"kubernetes.io/projected/3b377cbc-6859-4ae1-bde2-bac7680bb7d2-kube-api-access-cklx9\") pod \"payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn\" (UID: \"3b377cbc-6859-4ae1-bde2-bac7680bb7d2\") " pod="openshift-ingress/payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn" Oct 30 21:42:32.518121 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.518074 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcbbd3b-rh46j" event={"ID":"0015a27a-eba7-4ed4-8159-4dad866c5211","Type":"ContainerStarted","Data":"f14edb308a8ac4b2316f6d76d961eb921e7913510de75ddb0e26b7756d12895c"} Oct 30 21:42:32.518318 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.518303 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcbbd3b-rh46j" event={"ID":"0015a27a-eba7-4ed4-8159-4dad866c5211","Type":"ContainerStarted","Data":"65e1909d010de01a0b2e4cd33755b1a8304b2b9ab01d4cddf98d3a9335e7d312"} Oct 30 21:42:32.522536 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.522505 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" event={"ID":"c2c309a8-0866-464c-be99-67d454090ea6","Type":"ContainerStarted","Data":"87df9029cf7c4d8172e4dfe8c2b274c864b8c83ebb78f0cc31442563b4b7df4c"} Oct 30 21:42:32.522714 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.522624 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" event={"ID":"c2c309a8-0866-464c-be99-67d454090ea6","Type":"ContainerStarted","Data":"adcf86b79f183c6ebdea09c8ced257baedb375074cc64baa1c331e45274461cb"} Oct 30 21:42:32.543282 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.543232 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcbbd3b-rh46j" podStartSLOduration=2.543216936 podStartE2EDuration="2.543216936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:32.542480094 +0000 UTC m=+821.138296755" watchObservedRunningTime="2026-10-30 21:42:32.543216936 +0000 UTC m=+821.139033598" Oct 30 21:42:32.683276 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.683171 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:32.693942 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.693920 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:32.711215 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.711195 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s" Oct 30 21:42:32.775245 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.775214 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn" Oct 30 21:42:32.788954 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.788859 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:32.956879 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.938404 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:32.977783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:32.961924 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" podStartSLOduration=2.9619013990000003 podStartE2EDuration="2.961901399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:32.566931224 +0000 UTC m=+821.162747885" watchObservedRunningTime="2026-10-30 21:42:32.961901399 +0000 UTC m=+821.557718060" Oct 30 21:42:33.061224 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.056624 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f722129-qhn4n"] Oct 30 21:42:33.077128 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.077101 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f722129-qhn4n" Oct 30 21:42:33.091160 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.090555 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f722129-qhn4n"] Oct 30 21:42:33.112901 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.112857 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da8e5abc-c29b-49ac-b028-b29521dd8a94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f722129-qhn4n\" (UID: \"da8e5abc-c29b-49ac-b028-b29521dd8a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f722129-qhn4n" Oct 30 21:42:33.113075 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.113053 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da8e5abc-c29b-49ac-b028-b29521dd8a94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f722129-qhn4n\" (UID: \"da8e5abc-c29b-49ac-b028-b29521dd8a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f722129-qhn4n" Oct 30 21:42:33.113158 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.113141 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nfv8\" (UniqueName: \"kubernetes.io/projected/da8e5abc-c29b-49ac-b028-b29521dd8a94-kube-api-access-2nfv8\") pod \"maas-api-revoke-sub-7f722129-qhn4n\" (UID: \"da8e5abc-c29b-49ac-b028-b29521dd8a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f722129-qhn4n" Oct 30 21:42:33.121065 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.121043 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s"] Oct 30 21:42:33.214694 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.214587 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da8e5abc-c29b-49ac-b028-b29521dd8a94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f722129-qhn4n\" (UID: \"da8e5abc-c29b-49ac-b028-b29521dd8a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f722129-qhn4n" Oct 30 21:42:33.214906 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.214812 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nfv8\" (UniqueName: \"kubernetes.io/projected/da8e5abc-c29b-49ac-b028-b29521dd8a94-kube-api-access-2nfv8\") pod \"maas-api-revoke-sub-7f722129-qhn4n\" (UID: \"da8e5abc-c29b-49ac-b028-b29521dd8a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f722129-qhn4n" Oct 30 21:42:33.215060 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.215031 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da8e5abc-c29b-49ac-b028-b29521dd8a94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f722129-qhn4n\" (UID: \"da8e5abc-c29b-49ac-b028-b29521dd8a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f722129-qhn4n" Oct 30 21:42:33.216092 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.216066 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da8e5abc-c29b-49ac-b028-b29521dd8a94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f722129-qhn4n\" (UID: \"da8e5abc-c29b-49ac-b028-b29521dd8a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f722129-qhn4n" Oct 30 21:42:33.217857 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.217837 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da8e5abc-c29b-49ac-b028-b29521dd8a94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f722129-qhn4n\" (UID: \"da8e5abc-c29b-49ac-b028-b29521dd8a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f722129-qhn4n" Oct 30 21:42:33.229526 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.229419 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nfv8\" (UniqueName: \"kubernetes.io/projected/da8e5abc-c29b-49ac-b028-b29521dd8a94-kube-api-access-2nfv8\") pod \"maas-api-revoke-sub-7f722129-qhn4n\" (UID: \"da8e5abc-c29b-49ac-b028-b29521dd8a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f722129-qhn4n" Oct 30 21:42:33.350777 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:42:33.350730 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b377cbc_6859_4ae1_bde2_bac7680bb7d2.slice/crio-c85ad5a6b990b75413ed38c046dc0585ec9fff6a874a15781607e3ac3268d99b WatchSource:0}: Error finding container c85ad5a6b990b75413ed38c046dc0585ec9fff6a874a15781607e3ac3268d99b: Status 404 returned error can't find the container with id c85ad5a6b990b75413ed38c046dc0585ec9fff6a874a15781607e3ac3268d99b Oct 30 21:42:33.353785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.353742 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn"] Oct 30 21:42:33.370525 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.370499 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-911341b8-dx5th"] Oct 30 21:42:33.378871 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.378848 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-911341b8-dx5th" Oct 30 21:42:33.386423 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.386394 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-911341b8-dx5th"] Oct 30 21:42:33.419319 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.419289 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675ee6d3-1a83-4ab5-a258-d8445819f14b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-911341b8-dx5th\" (UID: \"675ee6d3-1a83-4ab5-a258-d8445819f14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-911341b8-dx5th" Oct 30 21:42:33.419844 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.419552 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lxw6\" (UniqueName: \"kubernetes.io/projected/675ee6d3-1a83-4ab5-a258-d8445819f14b-kube-api-access-7lxw6\") pod \"maas-api-revoke-sub-911341b8-dx5th\" (UID: \"675ee6d3-1a83-4ab5-a258-d8445819f14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-911341b8-dx5th" Oct 30 21:42:33.419844 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.419708 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675ee6d3-1a83-4ab5-a258-d8445819f14b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-911341b8-dx5th\" (UID: \"675ee6d3-1a83-4ab5-a258-d8445819f14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-911341b8-dx5th" Oct 30 21:42:33.448965 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.448935 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f722129-qhn4n" Oct 30 21:42:33.524783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.522242 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675ee6d3-1a83-4ab5-a258-d8445819f14b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-911341b8-dx5th\" (UID: \"675ee6d3-1a83-4ab5-a258-d8445819f14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-911341b8-dx5th" Oct 30 21:42:33.524783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.522487 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lxw6\" (UniqueName: \"kubernetes.io/projected/675ee6d3-1a83-4ab5-a258-d8445819f14b-kube-api-access-7lxw6\") pod \"maas-api-revoke-sub-911341b8-dx5th\" (UID: \"675ee6d3-1a83-4ab5-a258-d8445819f14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-911341b8-dx5th" Oct 30 21:42:33.524783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.522617 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675ee6d3-1a83-4ab5-a258-d8445819f14b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-911341b8-dx5th\" (UID: \"675ee6d3-1a83-4ab5-a258-d8445819f14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-911341b8-dx5th" Oct 30 21:42:33.524783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.523420 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675ee6d3-1a83-4ab5-a258-d8445819f14b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-911341b8-dx5th\" (UID: \"675ee6d3-1a83-4ab5-a258-d8445819f14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-911341b8-dx5th" Oct 30 21:42:33.526346 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.526301 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675ee6d3-1a83-4ab5-a258-d8445819f14b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-911341b8-dx5th\" (UID: \"675ee6d3-1a83-4ab5-a258-d8445819f14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-911341b8-dx5th" Oct 30 21:42:33.538074 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.538050 5801 generic.go:356] "Generic (PLEG): container finished" podID="0015a27a-eba7-4ed4-8159-4dad866c5211" containerID="f14edb308a8ac4b2316f6d76d961eb921e7913510de75ddb0e26b7756d12895c" exitCode=0 Oct 30 21:42:33.538172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.538144 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcbbd3b-rh46j" event={"ID":"0015a27a-eba7-4ed4-8159-4dad866c5211","Type":"ContainerDied","Data":"f14edb308a8ac4b2316f6d76d961eb921e7913510de75ddb0e26b7756d12895c"} Oct 30 21:42:33.538405 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.538379 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lxw6\" (UniqueName: \"kubernetes.io/projected/675ee6d3-1a83-4ab5-a258-d8445819f14b-kube-api-access-7lxw6\") pod \"maas-api-revoke-sub-911341b8-dx5th\" (UID: \"675ee6d3-1a83-4ab5-a258-d8445819f14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-911341b8-dx5th" Oct 30 21:42:33.540652 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.540629 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s" event={"ID":"6f0d6f19-bad5-415d-8230-125bf39ae64f","Type":"ContainerStarted","Data":"a2630c77f57453a2852a6b6be603af9b7e0e28a866502ed8aeec3acba24f7be5"} Oct 30 21:42:33.540783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.540669 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s" event={"ID":"6f0d6f19-bad5-415d-8230-125bf39ae64f","Type":"ContainerStarted","Data":"4849012e02f4d5653f9f49e7f404e2ea9fbb4813ed10eca0980bef3b1fbaeb84"} Oct 30 21:42:33.540783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.540695 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s" Oct 30 21:42:33.542926 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.542904 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn" event={"ID":"3b377cbc-6859-4ae1-bde2-bac7680bb7d2","Type":"ContainerStarted","Data":"13fccb87a2effea7bbe66e6253bf51b35a565118d7747ec3d92a275de215b187"} Oct 30 21:42:33.543012 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.542942 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn" event={"ID":"3b377cbc-6859-4ae1-bde2-bac7680bb7d2","Type":"ContainerStarted","Data":"c85ad5a6b990b75413ed38c046dc0585ec9fff6a874a15781607e3ac3268d99b"} Oct 30 21:42:33.577915 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.577865 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s" podStartSLOduration=1.5778511119999998 podStartE2EDuration="1.577851112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:33.574107038 +0000 UTC m=+822.169923726" watchObservedRunningTime="2026-10-30 21:42:33.577851112 +0000 UTC m=+822.173667793" Oct 30 21:42:33.594376 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.594335 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn" podStartSLOduration=1.594320966 podStartE2EDuration="1.594320966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:33.592287279 +0000 UTC m=+822.188103940" watchObservedRunningTime="2026-10-30 21:42:33.594320966 +0000 UTC m=+822.190137635" Oct 30 21:42:33.699387 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.699358 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:33.720882 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.720729 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-911341b8-dx5th" Oct 30 21:42:33.760789 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:42:33.760722 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8e5abc_c29b_49ac_b028_b29521dd8a94.slice/crio-5fae7a923e1d4fac4179a10ac764ccdcd0826232f2e0ce04277774975864471b WatchSource:0}: Error finding container 5fae7a923e1d4fac4179a10ac764ccdcd0826232f2e0ce04277774975864471b: Status 404 returned error can't find the container with id 5fae7a923e1d4fac4179a10ac764ccdcd0826232f2e0ce04277774975864471b Oct 30 21:42:33.763302 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:33.763270 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f722129-qhn4n"] Oct 30 21:42:34.044806 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:42:34.044745 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675ee6d3_1a83_4ab5_a258_d8445819f14b.slice/crio-839361a8d7a607e53873f12af1c1fde69fae58c3e44bdd2c79d90fc8d9a04023 WatchSource:0}: Error finding container 839361a8d7a607e53873f12af1c1fde69fae58c3e44bdd2c79d90fc8d9a04023: Status 404 returned error can't find the container with id 839361a8d7a607e53873f12af1c1fde69fae58c3e44bdd2c79d90fc8d9a04023 Oct 30 21:42:34.045855 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.045827 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-911341b8-dx5th"] Oct 30 21:42:34.106708 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.106671 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:34.535907 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.535870 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:34.554709 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.554676 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-911341b8-dx5th" event={"ID":"675ee6d3-1a83-4ab5-a258-d8445819f14b","Type":"ContainerStarted","Data":"deda7dde7953fdfda55687a3c9e74f6054ee46197e740a0d2b8ec49d2a000472"} Oct 30 21:42:34.555392 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.554737 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-911341b8-dx5th" event={"ID":"675ee6d3-1a83-4ab5-a258-d8445819f14b","Type":"ContainerStarted","Data":"839361a8d7a607e53873f12af1c1fde69fae58c3e44bdd2c79d90fc8d9a04023"} Oct 30 21:42:34.557093 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.557062 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f722129-qhn4n" event={"ID":"da8e5abc-c29b-49ac-b028-b29521dd8a94","Type":"ContainerStarted","Data":"b1eef8fb3557ad3fafa3e7cc01e7d49e006ce31f646d412db1eb642db4732d3e"} Oct 30 21:42:34.557226 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.557118 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f722129-qhn4n" event={"ID":"da8e5abc-c29b-49ac-b028-b29521dd8a94","Type":"ContainerStarted","Data":"5fae7a923e1d4fac4179a10ac764ccdcd0826232f2e0ce04277774975864471b"} Oct 30 21:42:34.557522 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.557497 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:34.557816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.557793 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn" Oct 30 21:42:34.558465 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.558446 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:42:34.584841 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.584787 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-911341b8-dx5th" podStartSLOduration=1.5847693889999999 podStartE2EDuration="1.584769389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:34.57426643 +0000 UTC m=+823.170083099" watchObservedRunningTime="2026-10-30 21:42:34.584769389 +0000 UTC m=+823.180586044" Oct 30 21:42:34.594394 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.594345 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f722129-qhn4n" podStartSLOduration=2.594332344 podStartE2EDuration="2.594332344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:34.593208873 +0000 UTC m=+823.189025554" watchObservedRunningTime="2026-10-30 21:42:34.594332344 +0000 UTC m=+823.190149004" Oct 30 21:42:34.643230 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.643156 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:34.752467 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.752442 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcbbd3b-rh46j" Oct 30 21:42:34.862346 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.862317 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0015a27a-eba7-4ed4-8159-4dad866c5211-kube-api-access-t266q\" (UniqueName: \"kubernetes.io/projected/0015a27a-eba7-4ed4-8159-4dad866c5211-kube-api-access-t266q\") pod \"0015a27a-eba7-4ed4-8159-4dad866c5211\" (UID: \"0015a27a-eba7-4ed4-8159-4dad866c5211\") " Oct 30 21:42:34.862346 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.862448 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0015a27a-eba7-4ed4-8159-4dad866c5211-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0015a27a-eba7-4ed4-8159-4dad866c5211-maas-api-openshift-ca\") pod \"0015a27a-eba7-4ed4-8159-4dad866c5211\" (UID: \"0015a27a-eba7-4ed4-8159-4dad866c5211\") " Oct 30 21:42:34.862854 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.862637 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0015a27a-eba7-4ed4-8159-4dad866c5211-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0015a27a-eba7-4ed4-8159-4dad866c5211-maas-api-xks-ca\") pod \"0015a27a-eba7-4ed4-8159-4dad866c5211\" (UID: \"0015a27a-eba7-4ed4-8159-4dad866c5211\") " Oct 30 21:42:34.863348 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.863317 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0015a27a-eba7-4ed4-8159-4dad866c5211-maas-api-openshift-ca" pod "0015a27a-eba7-4ed4-8159-4dad866c5211" (UID: "0015a27a-eba7-4ed4-8159-4dad866c5211"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:34.865445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.865422 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0015a27a-eba7-4ed4-8159-4dad866c5211-maas-api-xks-ca" pod "0015a27a-eba7-4ed4-8159-4dad866c5211" (UID: "0015a27a-eba7-4ed4-8159-4dad866c5211"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:42:34.865586 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.865568 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0015a27a-eba7-4ed4-8159-4dad866c5211-kube-api-access-t266q" pod "0015a27a-eba7-4ed4-8159-4dad866c5211" (UID: "0015a27a-eba7-4ed4-8159-4dad866c5211"). InnerVolumeSpecName "kube-api-access-t266q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:34.914647 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.914526 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:34.965578 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.965543 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0015a27a-eba7-4ed4-8159-4dad866c5211-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:34.965578 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.965579 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t266q\" (UniqueName: \"kubernetes.io/projected/0015a27a-eba7-4ed4-8159-4dad866c5211-kube-api-access-t266q\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:34.965854 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.965612 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0015a27a-eba7-4ed4-8159-4dad866c5211-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:34.977806 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.977772 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx"] Oct 30 21:42:34.981110 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.981084 5801 cpu_manager.go:414] "containerMap: removing container" podUID="0015a27a-eba7-4ed4-8159-4dad866c5211" containerName="revoke-keys" Oct 30 21:42:34.981110 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.981107 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0015a27a-eba7-4ed4-8159-4dad866c5211" containerName="revoke-keys" Oct 30 21:42:34.981341 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.981333 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="0015a27a-eba7-4ed4-8159-4dad866c5211" containerName="revoke-keys" Oct 30 21:42:34.986951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.986830 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx" Oct 30 21:42:34.988804 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.988777 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx"] Oct 30 21:42:34.996913 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.996887 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-938e67d1-dockercfg-stx79\"" Oct 30 21:42:34.997051 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:34.996975 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-938e67d1\"" Oct 30 21:42:35.019144 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:35.019110 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:35.067381 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:35.067347 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64qbc\" (UniqueName: \"kubernetes.io/projected/516bdeb1-af0e-44c0-9187-20dd69d16099-kube-api-access-64qbc\") pod \"payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx\" (UID: \"516bdeb1-af0e-44c0-9187-20dd69d16099\") " pod="openshift-ingress/payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx" Oct 30 21:42:35.067582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:35.067560 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/516bdeb1-af0e-44c0-9187-20dd69d16099-plugins-config-volume\") pod \"payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx\" (UID: \"516bdeb1-af0e-44c0-9187-20dd69d16099\") " pod="openshift-ingress/payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx" Oct 30 21:42:35.118485 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:35.118456 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:35.208858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:35.195136 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/516bdeb1-af0e-44c0-9187-20dd69d16099-plugins-config-volume\") pod \"payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx\" (UID: \"516bdeb1-af0e-44c0-9187-20dd69d16099\") " pod="openshift-ingress/payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx" Oct 30 21:42:35.208858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:35.196235 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/516bdeb1-af0e-44c0-9187-20dd69d16099-plugins-config-volume\") pod \"payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx\" (UID: \"516bdeb1-af0e-44c0-9187-20dd69d16099\") " pod="openshift-ingress/payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx" Oct 30 21:42:35.226616 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:35.226055 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64qbc\" (UniqueName: \"kubernetes.io/projected/516bdeb1-af0e-44c0-9187-20dd69d16099-kube-api-access-64qbc\") pod \"payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx\" (UID: \"516bdeb1-af0e-44c0-9187-20dd69d16099\") " pod="openshift-ingress/payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx" Oct 30 21:42:35.297784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:35.294786 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64qbc\" (UniqueName: \"kubernetes.io/projected/516bdeb1-af0e-44c0-9187-20dd69d16099-kube-api-access-64qbc\") pod \"payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx\" (UID: \"516bdeb1-af0e-44c0-9187-20dd69d16099\") " pod="openshift-ingress/payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx" Oct 30 21:42:35.318315 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:35.318044 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx" Oct 30 21:42:35.593740 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:35.593242 5801 generic.go:356] "Generic (PLEG): container finished" podID="675ee6d3-1a83-4ab5-a258-d8445819f14b" containerID="deda7dde7953fdfda55687a3c9e74f6054ee46197e740a0d2b8ec49d2a000472" exitCode=0 Oct 30 21:42:35.593740 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:35.593679 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-911341b8-dx5th" event={"ID":"675ee6d3-1a83-4ab5-a258-d8445819f14b","Type":"ContainerDied","Data":"deda7dde7953fdfda55687a3c9e74f6054ee46197e740a0d2b8ec49d2a000472"} Oct 30 21:42:35.598349 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:35.597963 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcbbd3b-rh46j" event={"ID":"0015a27a-eba7-4ed4-8159-4dad866c5211","Type":"ContainerDied","Data":"65e1909d010de01a0b2e4cd33755b1a8304b2b9ab01d4cddf98d3a9335e7d312"} Oct 30 21:42:35.598349 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:35.597998 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65e1909d010de01a0b2e4cd33755b1a8304b2b9ab01d4cddf98d3a9335e7d312" Oct 30 21:42:35.598349 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:35.598058 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcbbd3b-rh46j" Oct 30 21:42:35.603966 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:35.602050 5801 generic.go:356] "Generic (PLEG): container finished" podID="da8e5abc-c29b-49ac-b028-b29521dd8a94" containerID="b1eef8fb3557ad3fafa3e7cc01e7d49e006ce31f646d412db1eb642db4732d3e" exitCode=0 Oct 30 21:42:35.603966 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:35.603195 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f722129-qhn4n" event={"ID":"da8e5abc-c29b-49ac-b028-b29521dd8a94","Type":"ContainerDied","Data":"b1eef8fb3557ad3fafa3e7cc01e7d49e006ce31f646d412db1eb642db4732d3e"} Oct 30 21:42:35.696728 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:42:35.696696 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod516bdeb1_af0e_44c0_9187_20dd69d16099.slice/crio-7febf404bf0bb6d80dd27a85d187bd69ddc519929834a5bb8d7afd0c7f9d6eac WatchSource:0}: Error finding container 7febf404bf0bb6d80dd27a85d187bd69ddc519929834a5bb8d7afd0c7f9d6eac: Status 404 returned error can't find the container with id 7febf404bf0bb6d80dd27a85d187bd69ddc519929834a5bb8d7afd0c7f9d6eac Oct 30 21:42:35.699446 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:35.699420 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx"] Oct 30 21:42:36.196629 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:36.196589 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:36.342440 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:36.342394 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:36.459268 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:36.459143 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:36.757833 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:36.753823 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx" event={"ID":"516bdeb1-af0e-44c0-9187-20dd69d16099","Type":"ContainerStarted","Data":"efe36aadc803f54e7706139ee3366f5f6eac6058fa800efa21743f3311b7c75d"} Oct 30 21:42:36.757833 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:36.753919 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx" event={"ID":"516bdeb1-af0e-44c0-9187-20dd69d16099","Type":"ContainerStarted","Data":"7febf404bf0bb6d80dd27a85d187bd69ddc519929834a5bb8d7afd0c7f9d6eac"} Oct 30 21:42:36.813502 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:36.768469 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx" Oct 30 21:42:36.845777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:36.841796 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx" podStartSLOduration=2.841774068 podStartE2EDuration="2.841774068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:36.78407537 +0000 UTC m=+825.379892029" watchObservedRunningTime="2026-10-30 21:42:36.841774068 +0000 UTC m=+825.437590732" Oct 30 21:42:37.237896 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:37.237849 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:37.269913 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:37.269864 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:37.289892 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:37.289846 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:42:37.332788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:37.324320 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:37.372881 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:37.372837 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:37.415175 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:37.415132 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:37.440901 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:37.440853 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:37.779109 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:37.777994 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:42:37.787026 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:37.786990 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:42:37.808948 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:37.808903 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:37.875887 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:37.875842 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:42:38.074942 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.074892 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:38.183210 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.075477 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:42:38.255614 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.254383 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:38.263842 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.263810 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:42:38.318779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.316467 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:38.318779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.316835 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:42:38.318779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.317338 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:42:38.318779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.317391 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:42:38.396832 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.341097 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:42:38.396832 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.341643 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:42:38.451776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.426860 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:38.485977 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.484802 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:42:38.486470 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.484835 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:42:38.618359 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.558540 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:38.666830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.643157 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:38.773779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.771482 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:42:38.928866 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.808055 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:42:38.928866 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.808301 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:42:38.928866 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.811114 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:38.928866 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.899396 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:42:38.928866 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.903235 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded" Oct 30 21:42:38.928866 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.909232 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:42:38.928866 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.919967 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:42:38.928866 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.920063 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:42:38.928866 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:38.920125 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:42:39.164779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:39.154791 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:42:39.164779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:39.155394 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:42:39.241832 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:39.220641 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:42:39.241832 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:39.237196 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:42:39.264698 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:39.264649 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:39.632867 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:39.384122 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:39.632867 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:39.339274 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:42:39.632867 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:39.360581 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:39.632867 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:39.336355 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:39.632867 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:39.544680 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:42:39.749779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:39.730125 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:39.758850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:39.758791 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:42:39.861309 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:39.861253 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:39.861975 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:39.861943 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:42:39.862347 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:39.862314 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:42:39.921777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:39.919919 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:42:39.921777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:39.920115 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:40.198828 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:40.159834 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:42:40.230880 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:40.230829 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:40.437784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:40.405913 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:40.437784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:40.406213 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:40.437784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:40.406664 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:40.781991 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:40.781939 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:40.789775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:40.782572 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:40.929790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:40.928845 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:40.933784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:40.930345 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:41.040837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:41.021117 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:42:41.050895 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:41.042708 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" podUID="7e0434b3-6ab0-42c5-9c5b-e4592daadca7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded" Oct 30 21:42:41.057908 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:41.051691 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:41.065801 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:41.051737 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" podUID="b6307469-5f7d-41d9-9490-e8381ff9bbfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded" Oct 30 21:42:41.288310 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:41.288264 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:41.392776 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:42:41.390222 5801 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.845s" Oct 30 21:42:41.392776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:41.390308 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx" Oct 30 21:42:41.568824 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:41.294086 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:42:41.568824 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:41.051785 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:41.568824 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:41.522904 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:41.568824 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:41.523219 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded" Oct 30 21:42:41.568824 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:41.524506 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:42:41.568824 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:41.527979 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:41.568824 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:41.545714 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:42:41.802866 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:41.785272 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:41.977777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:41.913900 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:42:41.977777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:41.932637 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:41.977777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:41.932723 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:42:42.045776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:41.999546 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-911341b8-dx5th" event={"ID":"675ee6d3-1a83-4ab5-a258-d8445819f14b","Type":"ContainerDied","Data":"839361a8d7a607e53873f12af1c1fde69fae58c3e44bdd2c79d90fc8d9a04023"} Oct 30 21:42:42.045776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:41.999616 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="839361a8d7a607e53873f12af1c1fde69fae58c3e44bdd2c79d90fc8d9a04023" Oct 30 21:42:42.045776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.008822 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-911341b8-dx5th" Oct 30 21:42:42.045776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.020880 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f722129-qhn4n" Oct 30 21:42:42.045776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.022602 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:42:42.045776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.033819 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da8e5abc-c29b-49ac-b028-b29521dd8a94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da8e5abc-c29b-49ac-b028-b29521dd8a94-maas-api-xks-ca\") pod \"da8e5abc-c29b-49ac-b028-b29521dd8a94\" (UID: \"da8e5abc-c29b-49ac-b028-b29521dd8a94\") " Oct 30 21:42:42.045776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.033880 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da8e5abc-c29b-49ac-b028-b29521dd8a94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da8e5abc-c29b-49ac-b028-b29521dd8a94-maas-api-openshift-ca\") pod \"da8e5abc-c29b-49ac-b028-b29521dd8a94\" (UID: \"da8e5abc-c29b-49ac-b028-b29521dd8a94\") " Oct 30 21:42:42.045776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.033988 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da8e5abc-c29b-49ac-b028-b29521dd8a94-kube-api-access-2nfv8\" (UniqueName: \"kubernetes.io/projected/da8e5abc-c29b-49ac-b028-b29521dd8a94-kube-api-access-2nfv8\") pod \"da8e5abc-c29b-49ac-b028-b29521dd8a94\" (UID: \"da8e5abc-c29b-49ac-b028-b29521dd8a94\") " Oct 30 21:42:42.045776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.034021 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/675ee6d3-1a83-4ab5-a258-d8445819f14b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675ee6d3-1a83-4ab5-a258-d8445819f14b-maas-api-openshift-ca\") pod \"675ee6d3-1a83-4ab5-a258-d8445819f14b\" (UID: \"675ee6d3-1a83-4ab5-a258-d8445819f14b\") " Oct 30 21:42:42.045776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.034165 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/675ee6d3-1a83-4ab5-a258-d8445819f14b-kube-api-access-7lxw6\" (UniqueName: \"kubernetes.io/projected/675ee6d3-1a83-4ab5-a258-d8445819f14b-kube-api-access-7lxw6\") pod \"675ee6d3-1a83-4ab5-a258-d8445819f14b\" (UID: \"675ee6d3-1a83-4ab5-a258-d8445819f14b\") " Oct 30 21:42:42.045776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.034242 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/675ee6d3-1a83-4ab5-a258-d8445819f14b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675ee6d3-1a83-4ab5-a258-d8445819f14b-maas-api-xks-ca\") pod \"675ee6d3-1a83-4ab5-a258-d8445819f14b\" (UID: \"675ee6d3-1a83-4ab5-a258-d8445819f14b\") " Oct 30 21:42:42.072821 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.048202 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:42.073452 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.068569 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:42.074771 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.068621 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f722129-qhn4n" event={"ID":"da8e5abc-c29b-49ac-b028-b29521dd8a94","Type":"ContainerDied","Data":"5fae7a923e1d4fac4179a10ac764ccdcd0826232f2e0ce04277774975864471b"} Oct 30 21:42:42.074971 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.074955 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fae7a923e1d4fac4179a10ac764ccdcd0826232f2e0ce04277774975864471b" Oct 30 21:42:42.075052 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.068880 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da8e5abc-c29b-49ac-b028-b29521dd8a94-maas-api-openshift-ca" pod "da8e5abc-c29b-49ac-b028-b29521dd8a94" (UID: "da8e5abc-c29b-49ac-b028-b29521dd8a94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:42.114033 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.113980 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/675ee6d3-1a83-4ab5-a258-d8445819f14b-maas-api-openshift-ca" pod "675ee6d3-1a83-4ab5-a258-d8445819f14b" (UID: "675ee6d3-1a83-4ab5-a258-d8445819f14b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:42.152994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.152952 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da8e5abc-c29b-49ac-b028-b29521dd8a94-maas-api-xks-ca" pod "da8e5abc-c29b-49ac-b028-b29521dd8a94" (UID: "da8e5abc-c29b-49ac-b028-b29521dd8a94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:42:42.229779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.204305 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da8e5abc-c29b-49ac-b028-b29521dd8a94-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:42.229779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.204341 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da8e5abc-c29b-49ac-b028-b29521dd8a94-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:42.229779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.204357 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675ee6d3-1a83-4ab5-a258-d8445819f14b-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:42.256778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.255942 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/675ee6d3-1a83-4ab5-a258-d8445819f14b-kube-api-access-7lxw6" pod "675ee6d3-1a83-4ab5-a258-d8445819f14b" (UID: "675ee6d3-1a83-4ab5-a258-d8445819f14b"). InnerVolumeSpecName "kube-api-access-7lxw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:42.256778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.256048 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/675ee6d3-1a83-4ab5-a258-d8445819f14b-maas-api-xks-ca" pod "675ee6d3-1a83-4ab5-a258-d8445819f14b" (UID: "675ee6d3-1a83-4ab5-a258-d8445819f14b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:42:42.256778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.256115 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8e5abc-c29b-49ac-b028-b29521dd8a94-kube-api-access-2nfv8" pod "da8e5abc-c29b-49ac-b028-b29521dd8a94" (UID: "da8e5abc-c29b-49ac-b028-b29521dd8a94"). InnerVolumeSpecName "kube-api-access-2nfv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:42.256778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.256348 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:42.307775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.307298 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lxw6\" (UniqueName: \"kubernetes.io/projected/675ee6d3-1a83-4ab5-a258-d8445819f14b-kube-api-access-7lxw6\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:42.307775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.307329 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675ee6d3-1a83-4ab5-a258-d8445819f14b-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:42.307775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.307343 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nfv8\" (UniqueName: \"kubernetes.io/projected/da8e5abc-c29b-49ac-b028-b29521dd8a94-kube-api-access-2nfv8\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:42.381825 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.368223 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:42.483835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.483515 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:42.570779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.533844 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:42.570779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.534207 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:42:42.570779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.534476 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:42.681311 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.581972 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:42.681311 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.616830 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:42:42.764024 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.763975 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:42:42.764302 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:42.764257 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" podUID="321186ea-7520-430c-825b-b8129b422d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:43.117780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:43.115961 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:43.146781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:43.142907 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:43.146781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:43.144335 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": context deadline exceeded" Oct 30 21:42:43.193779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:43.192715 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:42:43.193779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:43.193355 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:42:43.194034 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:43.193900 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:42:43.224776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:43.195109 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:42:43.263780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:43.230227 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:42:43.263780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:43.230296 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" podUID="528a2150-a9c8-42d4-8203-113bc618287a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:43.263780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:43.252725 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:42:43.263780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:43.252818 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" podUID="0f258bc9-d2fd-446a-9154-36caf951e081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:43.313666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:43.313618 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:43.369019 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:43.368920 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:43.503973 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:43.503923 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:43.545907 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:43.545853 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:43.546561 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:43.546527 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:42:43.547079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:43.547051 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:43.634169 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:43.611736 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f722129-qhn4n" Oct 30 21:42:43.678776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:43.646338 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:43.737780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:43.627863 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sk2kw" podUID="faea70fa-2fa1-45b8-9409-72fb5d589cc3" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.25:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:43.799470 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:43.680724 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-911341b8-dx5th" Oct 30 21:42:43.799924 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:43.627809 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:44.088779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:44.074821 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:42:44.088779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:44.074884 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" podUID="6aaf74ef-1550-443a-a046-d08fc45e07f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:44.124334 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:44.124290 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:44.154778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:44.143508 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:44.220708 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:44.220260 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:44.260798 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:44.259741 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:42:44.260798 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:44.260130 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:42:44.260798 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:44.260471 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:42:44.262971 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:44.262945 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:42:44.290199 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:44.290164 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" podUID="b8cc8b5b-1443-40da-a41d-745fcba8e474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:44.379598 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:44.379504 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:44.552788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:44.527293 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:42:44.562811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:44.560951 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:42:44.562811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:44.561348 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:42:44.562811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:44.561622 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:42:44.738861 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:44.706986 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s" Oct 30 21:42:44.858782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:44.817120 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:44.902787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:44.838723 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:42:44.950781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:44.872810 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:45.030854 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:45.022855 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:45.142997 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:45.141952 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:45.190781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:45.190352 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 21:42:45.220211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:45.219999 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Oct 30 21:42:45.220946 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:45.220459 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:42:45.321831 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:45.270009 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:45.321831 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:45.295277 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded" Oct 30 21:42:45.321831 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:45.295378 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:42:45.321831 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:45.296534 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:42:45.401802 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:45.396841 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:45.642832 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:45.636936 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:45.684445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:45.684402 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:45.714776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:45.689580 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:42:45.833787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:45.821605 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:42:45.885911 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:45.834641 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:45.933163 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:45.930650 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:42:46.092928 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.092881 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:46.103772 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.103135 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:42:46.149095 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.135174 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:42:46.256933 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.256892 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:46.259500 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.259471 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:42:46.260572 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.257159 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:46.297774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.288255 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:42:46.349864 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.303947 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:46.350538 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.315486 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn" Oct 30 21:42:46.351905 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.292857 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:42:46.358068 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.358014 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:42:46.358207 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.341113 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:46.359654 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.359631 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:42:46.361377 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.361351 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:42:46.361468 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.361419 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:42:46.362664 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.362647 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:42:46.362956 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.362937 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:42:46.369437 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.367258 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:42:46.382452 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.381830 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:42:46.382641 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.382624 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:42:46.384852 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.384833 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:42:46.385148 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.385130 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:42:46.463987 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.457877 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:46.795049 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.794957 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769"] Oct 30 21:42:46.796102 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.796080 5801 cpu_manager.go:414] "containerMap: removing container" podUID="da8e5abc-c29b-49ac-b028-b29521dd8a94" containerName="revoke-keys" Oct 30 21:42:46.796190 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.796105 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da8e5abc-c29b-49ac-b028-b29521dd8a94" containerName="revoke-keys" Oct 30 21:42:46.796190 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.796139 5801 cpu_manager.go:414] "containerMap: removing container" podUID="675ee6d3-1a83-4ab5-a258-d8445819f14b" containerName="revoke-keys" Oct 30 21:42:46.796190 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.796147 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="675ee6d3-1a83-4ab5-a258-d8445819f14b" containerName="revoke-keys" Oct 30 21:42:46.796328 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.796315 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="675ee6d3-1a83-4ab5-a258-d8445819f14b" containerName="revoke-keys" Oct 30 21:42:46.796367 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.796347 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="da8e5abc-c29b-49ac-b028-b29521dd8a94" containerName="revoke-keys" Oct 30 21:42:46.802227 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.802204 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:46.812445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.812376 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769"] Oct 30 21:42:46.843906 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.843868 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 21:42:46.844106 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.843938 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 21:42:46.844201 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.844108 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-747gg\"" Oct 30 21:42:46.844201 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.844133 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 21:42:46.882771 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.882472 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jv769\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:46.882771 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.882521 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jv769\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:46.882771 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.882567 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jv769\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:46.882771 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.882701 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pksgp\" (UniqueName: \"kubernetes.io/projected/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-kube-api-access-pksgp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jv769\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:46.883130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.882855 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jv769\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:46.883130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.882949 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jv769\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:46.883130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.883051 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jv769\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:46.929095 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.929062 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:46.985525 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.984982 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pksgp\" (UniqueName: \"kubernetes.io/projected/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-kube-api-access-pksgp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jv769\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:46.985525 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.985114 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jv769\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:46.985525 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.985193 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jv769\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:46.985525 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.985293 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jv769\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:46.985525 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.985396 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jv769\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:46.985525 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.985420 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jv769\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:46.985525 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.985461 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jv769\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:46.986338 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.986316 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jv769\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:46.996252 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.995092 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jv769\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:47.001374 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:46.998663 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jv769\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:47.011723 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:47.002882 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jv769\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:47.011723 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:47.010294 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jv769\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:47.033357 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:47.033327 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pksgp\" (UniqueName: \"kubernetes.io/projected/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-kube-api-access-pksgp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jv769\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:47.037441 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:47.037410 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jv769\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:47.064711 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:47.064676 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:47.122013 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:47.121980 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:47.218412 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:47.218207 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:47.338799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:47.338675 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:47.503058 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:47.503018 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769"] Oct 30 21:42:47.503864 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:42:47.503830 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0e7692_fb76_4697_a947_a0ff3bfd72ba.slice/crio-e3bbd8206991b594bde7b13abaf5e26cac63c1e44dea6c6bc9d9d49b47321bc3 WatchSource:0}: Error finding container e3bbd8206991b594bde7b13abaf5e26cac63c1e44dea6c6bc9d9d49b47321bc3: Status 404 returned error can't find the container with id e3bbd8206991b594bde7b13abaf5e26cac63c1e44dea6c6bc9d9d49b47321bc3 Oct 30 21:42:48.239590 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:48.239551 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:48.337930 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:48.337633 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" event={"ID":"eb0e7692-fb76-4697-a947-a0ff3bfd72ba","Type":"ContainerStarted","Data":"e3bbd8206991b594bde7b13abaf5e26cac63c1e44dea6c6bc9d9d49b47321bc3"} Oct 30 21:42:49.051611 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:49.051567 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:49.223012 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:49.222976 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0edc820-2shwf"] Oct 30 21:42:49.229799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:49.229772 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0edc820-2shwf" Oct 30 21:42:49.233287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:49.233256 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0edc820-2shwf"] Oct 30 21:42:49.240567 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:49.235536 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:42:49.240567 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:49.235594 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:42:49.240567 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:49.237932 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:42:49.240567 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:49.238011 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:42:49.362282 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:49.362244 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ffb13b5-2e3b-4f20-a091-a9f24155c339-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0edc820-2shwf\" (UID: \"2ffb13b5-2e3b-4f20-a091-a9f24155c339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0edc820-2shwf" Oct 30 21:42:49.362485 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:49.362364 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ffb13b5-2e3b-4f20-a091-a9f24155c339-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0edc820-2shwf\" (UID: \"2ffb13b5-2e3b-4f20-a091-a9f24155c339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0edc820-2shwf" Oct 30 21:42:49.362485 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:49.362395 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzns5\" (UniqueName: \"kubernetes.io/projected/2ffb13b5-2e3b-4f20-a091-a9f24155c339-kube-api-access-bzns5\") pod \"maas-api-revoke-sub-b0edc820-2shwf\" (UID: \"2ffb13b5-2e3b-4f20-a091-a9f24155c339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0edc820-2shwf" Oct 30 21:42:49.464004 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:49.463973 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzns5\" (UniqueName: \"kubernetes.io/projected/2ffb13b5-2e3b-4f20-a091-a9f24155c339-kube-api-access-bzns5\") pod \"maas-api-revoke-sub-b0edc820-2shwf\" (UID: \"2ffb13b5-2e3b-4f20-a091-a9f24155c339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0edc820-2shwf" Oct 30 21:42:49.464371 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:49.464349 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ffb13b5-2e3b-4f20-a091-a9f24155c339-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0edc820-2shwf\" (UID: \"2ffb13b5-2e3b-4f20-a091-a9f24155c339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0edc820-2shwf" Oct 30 21:42:49.464470 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:49.464424 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ffb13b5-2e3b-4f20-a091-a9f24155c339-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0edc820-2shwf\" (UID: \"2ffb13b5-2e3b-4f20-a091-a9f24155c339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0edc820-2shwf" Oct 30 21:42:49.465118 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:49.465089 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ffb13b5-2e3b-4f20-a091-a9f24155c339-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0edc820-2shwf\" (UID: \"2ffb13b5-2e3b-4f20-a091-a9f24155c339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0edc820-2shwf" Oct 30 21:42:49.467241 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:49.467218 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ffb13b5-2e3b-4f20-a091-a9f24155c339-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0edc820-2shwf\" (UID: \"2ffb13b5-2e3b-4f20-a091-a9f24155c339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0edc820-2shwf" Oct 30 21:42:49.476895 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:49.476875 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzns5\" (UniqueName: \"kubernetes.io/projected/2ffb13b5-2e3b-4f20-a091-a9f24155c339-kube-api-access-bzns5\") pod \"maas-api-revoke-sub-b0edc820-2shwf\" (UID: \"2ffb13b5-2e3b-4f20-a091-a9f24155c339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0edc820-2shwf" Oct 30 21:42:49.549989 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:49.549958 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0edc820-2shwf" Oct 30 21:42:49.852517 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:49.852476 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:49.861284 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:49.861252 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0edc820-2shwf"] Oct 30 21:42:49.963842 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:49.961310 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:50.160006 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:50.159937 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:50.446211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:50.446091 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0edc820-2shwf" event={"ID":"2ffb13b5-2e3b-4f20-a091-a9f24155c339","Type":"ContainerStarted","Data":"8c8b4996f9874481677f82b29c48c908bec7e43188cb4f361b4339ccc4dbdebe"} Oct 30 21:42:50.446211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:50.446159 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0edc820-2shwf" event={"ID":"2ffb13b5-2e3b-4f20-a091-a9f24155c339","Type":"ContainerStarted","Data":"49267d58b9311687f9f9fb2118cbf533556ec61ee80ae486b2001294790a850c"} Oct 30 21:42:50.466089 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:50.466024 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0edc820-2shwf" podStartSLOduration=1.466006906 podStartE2EDuration="1.466006906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:50.46480917 +0000 UTC m=+839.060625832" watchObservedRunningTime="2026-10-30 21:42:50.466006906 +0000 UTC m=+839.061823566" Oct 30 21:42:50.732783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:50.732692 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:50.893033 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:50.893003 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:42:50.893133 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:50.893067 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:42:51.490773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:51.484688 5801 generic.go:356] "Generic (PLEG): container finished" podID="2ffb13b5-2e3b-4f20-a091-a9f24155c339" containerID="8c8b4996f9874481677f82b29c48c908bec7e43188cb4f361b4339ccc4dbdebe" exitCode=0 Oct 30 21:42:51.490773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:51.484723 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0edc820-2shwf" event={"ID":"2ffb13b5-2e3b-4f20-a091-a9f24155c339","Type":"ContainerDied","Data":"8c8b4996f9874481677f82b29c48c908bec7e43188cb4f361b4339ccc4dbdebe"} Oct 30 21:42:51.490773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:51.487163 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" event={"ID":"eb0e7692-fb76-4697-a947-a0ff3bfd72ba","Type":"ContainerStarted","Data":"e6840060f3c8074f923f7c1cfc3baf04a8486d5ef002143fbf1561e4edc81b1d"} Oct 30 21:42:51.490773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:51.487861 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:51.490773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:51.489884 5801 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-jv769 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:42:51.490773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:51.489925 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" podUID="eb0e7692-fb76-4697-a947-a0ff3bfd72ba" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:42:51.537067 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:51.537018 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" podStartSLOduration=2.151873819 podStartE2EDuration="5.537004736s" totalImagesPullingTime="3.385130917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:51.534975771 +0000 UTC m=+840.130792449" watchObservedRunningTime="2026-10-30 21:42:51.537004736 +0000 UTC m=+840.132821396" Oct 30 21:42:51.558876 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:51.558836 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:52.424997 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:52.424962 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:52.498477 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:52.498258 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:42:52.727396 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:52.727372 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0edc820-2shwf" Oct 30 21:42:52.843839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:52.843809 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ffb13b5-2e3b-4f20-a091-a9f24155c339-kube-api-access-bzns5\" (UniqueName: \"kubernetes.io/projected/2ffb13b5-2e3b-4f20-a091-a9f24155c339-kube-api-access-bzns5\") pod \"2ffb13b5-2e3b-4f20-a091-a9f24155c339\" (UID: \"2ffb13b5-2e3b-4f20-a091-a9f24155c339\") " Oct 30 21:42:52.844042 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:52.843985 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:52.844127 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:52.844063 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ffb13b5-2e3b-4f20-a091-a9f24155c339-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ffb13b5-2e3b-4f20-a091-a9f24155c339-maas-api-xks-ca\") pod \"2ffb13b5-2e3b-4f20-a091-a9f24155c339\" (UID: \"2ffb13b5-2e3b-4f20-a091-a9f24155c339\") " Oct 30 21:42:52.844251 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:52.844233 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ffb13b5-2e3b-4f20-a091-a9f24155c339-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ffb13b5-2e3b-4f20-a091-a9f24155c339-maas-api-openshift-ca\") pod \"2ffb13b5-2e3b-4f20-a091-a9f24155c339\" (UID: \"2ffb13b5-2e3b-4f20-a091-a9f24155c339\") " Oct 30 21:42:52.844580 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:52.844557 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ffb13b5-2e3b-4f20-a091-a9f24155c339-maas-api-openshift-ca" pod "2ffb13b5-2e3b-4f20-a091-a9f24155c339" (UID: "2ffb13b5-2e3b-4f20-a091-a9f24155c339"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:52.844914 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:52.844897 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ffb13b5-2e3b-4f20-a091-a9f24155c339-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:52.846141 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:52.846116 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ffb13b5-2e3b-4f20-a091-a9f24155c339-kube-api-access-bzns5" pod "2ffb13b5-2e3b-4f20-a091-a9f24155c339" (UID: "2ffb13b5-2e3b-4f20-a091-a9f24155c339"). InnerVolumeSpecName "kube-api-access-bzns5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:52.846240 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:52.846166 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ffb13b5-2e3b-4f20-a091-a9f24155c339-maas-api-xks-ca" pod "2ffb13b5-2e3b-4f20-a091-a9f24155c339" (UID: "2ffb13b5-2e3b-4f20-a091-a9f24155c339"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:42:52.946364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:52.946285 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzns5\" (UniqueName: \"kubernetes.io/projected/2ffb13b5-2e3b-4f20-a091-a9f24155c339-kube-api-access-bzns5\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:52.946364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:52.946313 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ffb13b5-2e3b-4f20-a091-a9f24155c339-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:53.251013 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:53.250932 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:53.510090 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:53.510011 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0edc820-2shwf" Oct 30 21:42:53.510492 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:53.510018 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0edc820-2shwf" event={"ID":"2ffb13b5-2e3b-4f20-a091-a9f24155c339","Type":"ContainerDied","Data":"49267d58b9311687f9f9fb2118cbf533556ec61ee80ae486b2001294790a850c"} Oct 30 21:42:53.510492 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:53.510193 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49267d58b9311687f9f9fb2118cbf533556ec61ee80ae486b2001294790a850c" Oct 30 21:42:53.840900 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:53.840863 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7a0e4ed-88dm7"] Oct 30 21:42:53.841787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:53.841735 5801 cpu_manager.go:414] "containerMap: removing container" podUID="2ffb13b5-2e3b-4f20-a091-a9f24155c339" containerName="revoke-keys" Oct 30 21:42:53.841787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:53.841770 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ffb13b5-2e3b-4f20-a091-a9f24155c339" containerName="revoke-keys" Oct 30 21:42:53.842542 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:53.842020 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ffb13b5-2e3b-4f20-a091-a9f24155c339" containerName="revoke-keys" Oct 30 21:42:53.847203 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:53.847182 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a0e4ed-88dm7" Oct 30 21:42:53.849849 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:53.849821 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7a0e4ed-88dm7"] Oct 30 21:42:53.850987 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:53.850963 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:42:53.854772 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:53.852015 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:42:53.854772 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:53.852097 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:42:53.854772 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:53.852102 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:42:53.968265 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:53.968232 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk4bb\" (UniqueName: \"kubernetes.io/projected/5c7884e4-268e-46b2-808e-178ee8dc3398-kube-api-access-mk4bb\") pod \"maas-api-revoke-sub-a7a0e4ed-88dm7\" (UID: \"5c7884e4-268e-46b2-808e-178ee8dc3398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a0e4ed-88dm7" Oct 30 21:42:53.968491 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:53.968475 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c7884e4-268e-46b2-808e-178ee8dc3398-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7a0e4ed-88dm7\" (UID: \"5c7884e4-268e-46b2-808e-178ee8dc3398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a0e4ed-88dm7" Oct 30 21:42:53.968571 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:53.968559 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c7884e4-268e-46b2-808e-178ee8dc3398-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7a0e4ed-88dm7\" (UID: \"5c7884e4-268e-46b2-808e-178ee8dc3398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a0e4ed-88dm7" Oct 30 21:42:54.047296 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:54.047248 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:54.070363 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:54.070327 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c7884e4-268e-46b2-808e-178ee8dc3398-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7a0e4ed-88dm7\" (UID: \"5c7884e4-268e-46b2-808e-178ee8dc3398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a0e4ed-88dm7" Oct 30 21:42:54.070545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:54.070389 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c7884e4-268e-46b2-808e-178ee8dc3398-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7a0e4ed-88dm7\" (UID: \"5c7884e4-268e-46b2-808e-178ee8dc3398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a0e4ed-88dm7" Oct 30 21:42:54.070545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:54.070520 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk4bb\" (UniqueName: \"kubernetes.io/projected/5c7884e4-268e-46b2-808e-178ee8dc3398-kube-api-access-mk4bb\") pod \"maas-api-revoke-sub-a7a0e4ed-88dm7\" (UID: \"5c7884e4-268e-46b2-808e-178ee8dc3398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a0e4ed-88dm7" Oct 30 21:42:54.071049 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:54.071014 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c7884e4-268e-46b2-808e-178ee8dc3398-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7a0e4ed-88dm7\" (UID: \"5c7884e4-268e-46b2-808e-178ee8dc3398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a0e4ed-88dm7" Oct 30 21:42:54.072805 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:54.072779 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c7884e4-268e-46b2-808e-178ee8dc3398-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7a0e4ed-88dm7\" (UID: \"5c7884e4-268e-46b2-808e-178ee8dc3398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a0e4ed-88dm7" Oct 30 21:42:54.084197 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:54.084167 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk4bb\" (UniqueName: \"kubernetes.io/projected/5c7884e4-268e-46b2-808e-178ee8dc3398-kube-api-access-mk4bb\") pod \"maas-api-revoke-sub-a7a0e4ed-88dm7\" (UID: \"5c7884e4-268e-46b2-808e-178ee8dc3398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a0e4ed-88dm7" Oct 30 21:42:54.165135 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:54.165047 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a0e4ed-88dm7" Oct 30 21:42:54.502701 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:54.500505 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7a0e4ed-88dm7"] Oct 30 21:42:54.502701 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:42:54.501947 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7884e4_268e_46b2_808e_178ee8dc3398.slice/crio-186c2c6a11cd681b9451b20874726e879ab739e0f1ebb6e559b5a15136f734f7 WatchSource:0}: Error finding container 186c2c6a11cd681b9451b20874726e879ab739e0f1ebb6e559b5a15136f734f7: Status 404 returned error can't find the container with id 186c2c6a11cd681b9451b20874726e879ab739e0f1ebb6e559b5a15136f734f7 Oct 30 21:42:54.521775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:54.521728 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a0e4ed-88dm7" event={"ID":"5c7884e4-268e-46b2-808e-178ee8dc3398","Type":"ContainerStarted","Data":"186c2c6a11cd681b9451b20874726e879ab739e0f1ebb6e559b5a15136f734f7"} Oct 30 21:42:54.724118 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:54.724082 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:54.832728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:54.832667 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:55.045941 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:55.045850 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:55.146102 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:55.146063 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:55.531798 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:55.531768 5801 generic.go:356] "Generic (PLEG): container finished" podID="5c7884e4-268e-46b2-808e-178ee8dc3398" containerID="099312abba90ca5fe5e9e8299dd92bdca25be2cca3be07fd6cd76ed1dd3c22ed" exitCode=0 Oct 30 21:42:55.532196 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:55.531860 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a0e4ed-88dm7" event={"ID":"5c7884e4-268e-46b2-808e-178ee8dc3398","Type":"ContainerDied","Data":"099312abba90ca5fe5e9e8299dd92bdca25be2cca3be07fd6cd76ed1dd3c22ed"} Oct 30 21:42:55.970333 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:55.970295 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:56.723131 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:56.723109 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a0e4ed-88dm7" Oct 30 21:42:56.761694 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:56.761661 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:56.833567 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:56.833531 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c7884e4-268e-46b2-808e-178ee8dc3398-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c7884e4-268e-46b2-808e-178ee8dc3398-maas-api-xks-ca\") pod \"5c7884e4-268e-46b2-808e-178ee8dc3398\" (UID: \"5c7884e4-268e-46b2-808e-178ee8dc3398\") " Oct 30 21:42:56.833774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:56.833665 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c7884e4-268e-46b2-808e-178ee8dc3398-kube-api-access-mk4bb\" (UniqueName: \"kubernetes.io/projected/5c7884e4-268e-46b2-808e-178ee8dc3398-kube-api-access-mk4bb\") pod \"5c7884e4-268e-46b2-808e-178ee8dc3398\" (UID: \"5c7884e4-268e-46b2-808e-178ee8dc3398\") " Oct 30 21:42:56.833936 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:56.833917 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c7884e4-268e-46b2-808e-178ee8dc3398-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c7884e4-268e-46b2-808e-178ee8dc3398-maas-api-openshift-ca\") pod \"5c7884e4-268e-46b2-808e-178ee8dc3398\" (UID: \"5c7884e4-268e-46b2-808e-178ee8dc3398\") " Oct 30 21:42:56.834294 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:56.834257 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c7884e4-268e-46b2-808e-178ee8dc3398-maas-api-openshift-ca" pod "5c7884e4-268e-46b2-808e-178ee8dc3398" (UID: "5c7884e4-268e-46b2-808e-178ee8dc3398"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:56.834742 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:56.834723 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c7884e4-268e-46b2-808e-178ee8dc3398-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:56.836273 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:56.836246 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c7884e4-268e-46b2-808e-178ee8dc3398-maas-api-xks-ca" pod "5c7884e4-268e-46b2-808e-178ee8dc3398" (UID: "5c7884e4-268e-46b2-808e-178ee8dc3398"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:42:56.836369 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:56.836283 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7884e4-268e-46b2-808e-178ee8dc3398-kube-api-access-mk4bb" pod "5c7884e4-268e-46b2-808e-178ee8dc3398" (UID: "5c7884e4-268e-46b2-808e-178ee8dc3398"). InnerVolumeSpecName "kube-api-access-mk4bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:56.936769 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:56.936723 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c7884e4-268e-46b2-808e-178ee8dc3398-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:56.937006 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:56.936792 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk4bb\" (UniqueName: \"kubernetes.io/projected/5c7884e4-268e-46b2-808e-178ee8dc3398-kube-api-access-mk4bb\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:42:57.451292 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:57.451258 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:57.551193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:57.551155 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:57.556077 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:57.556045 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a0e4ed-88dm7" Oct 30 21:42:57.556226 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:57.556046 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a0e4ed-88dm7" event={"ID":"5c7884e4-268e-46b2-808e-178ee8dc3398","Type":"ContainerDied","Data":"186c2c6a11cd681b9451b20874726e879ab739e0f1ebb6e559b5a15136f734f7"} Oct 30 21:42:57.556226 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:57.556181 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="186c2c6a11cd681b9451b20874726e879ab739e0f1ebb6e559b5a15136f734f7" Oct 30 21:42:58.172675 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:58.172640 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:58.990184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:58.990148 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:59.478209 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:59.478175 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:59.579115 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:59.579076 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:59.698927 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:59.698896 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p"] Oct 30 21:42:59.699531 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:59.699516 5801 cpu_manager.go:414] "containerMap: removing container" podUID="5c7884e4-268e-46b2-808e-178ee8dc3398" containerName="revoke-keys" Oct 30 21:42:59.699585 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:59.699532 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c7884e4-268e-46b2-808e-178ee8dc3398" containerName="revoke-keys" Oct 30 21:42:59.699672 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:59.699662 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c7884e4-268e-46b2-808e-178ee8dc3398" containerName="revoke-keys" Oct 30 21:42:59.703875 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:59.703859 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p" Oct 30 21:42:59.708252 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:59.708228 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:42:59.708522 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:59.708501 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:42:59.708886 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:59.708863 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:42:59.709010 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:59.708992 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:42:59.713227 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:59.713205 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p"] Oct 30 21:42:59.787592 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:59.787521 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:59.883594 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:59.883557 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8vp4\" (UniqueName: \"kubernetes.io/projected/968f2902-38b5-49f2-b219-89cf41af462d-kube-api-access-r8vp4\") pod \"maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p\" (UID: \"968f2902-38b5-49f2-b219-89cf41af462d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p" Oct 30 21:42:59.883807 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:59.883634 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/968f2902-38b5-49f2-b219-89cf41af462d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p\" (UID: \"968f2902-38b5-49f2-b219-89cf41af462d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p" Oct 30 21:42:59.883807 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:59.883745 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/968f2902-38b5-49f2-b219-89cf41af462d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p\" (UID: \"968f2902-38b5-49f2-b219-89cf41af462d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p" Oct 30 21:42:59.985983 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:59.985947 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8vp4\" (UniqueName: \"kubernetes.io/projected/968f2902-38b5-49f2-b219-89cf41af462d-kube-api-access-r8vp4\") pod \"maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p\" (UID: \"968f2902-38b5-49f2-b219-89cf41af462d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p" Oct 30 21:42:59.986210 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:59.986090 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/968f2902-38b5-49f2-b219-89cf41af462d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p\" (UID: \"968f2902-38b5-49f2-b219-89cf41af462d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p" Oct 30 21:42:59.986210 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:59.986128 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/968f2902-38b5-49f2-b219-89cf41af462d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p\" (UID: \"968f2902-38b5-49f2-b219-89cf41af462d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p" Oct 30 21:42:59.986793 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:59.986736 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/968f2902-38b5-49f2-b219-89cf41af462d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p\" (UID: \"968f2902-38b5-49f2-b219-89cf41af462d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p" Oct 30 21:42:59.988506 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:59.988488 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/968f2902-38b5-49f2-b219-89cf41af462d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p\" (UID: \"968f2902-38b5-49f2-b219-89cf41af462d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p" Oct 30 21:42:59.998034 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:42:59.998006 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8vp4\" (UniqueName: \"kubernetes.io/projected/968f2902-38b5-49f2-b219-89cf41af462d-kube-api-access-r8vp4\") pod \"maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p\" (UID: \"968f2902-38b5-49f2-b219-89cf41af462d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p" Oct 30 21:43:00.023063 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:00.023038 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p" Oct 30 21:43:00.279575 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:43:00.279538 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:43:00.309766 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:43:00.309721 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968f2902_38b5_49f2_b219_89cf41af462d.slice/crio-c9ae5fbf762bfc0c024e975d3095c778b73aae8da6dc4f7ad587a67af80b4f9f WatchSource:0}: Error finding container c9ae5fbf762bfc0c024e975d3095c778b73aae8da6dc4f7ad587a67af80b4f9f: Status 404 returned error can't find the container with id c9ae5fbf762bfc0c024e975d3095c778b73aae8da6dc4f7ad587a67af80b4f9f Oct 30 21:43:00.312218 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:00.312194 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p"] Oct 30 21:43:00.597405 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:00.588711 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p" event={"ID":"968f2902-38b5-49f2-b219-89cf41af462d","Type":"ContainerStarted","Data":"c9ae5fbf762bfc0c024e975d3095c778b73aae8da6dc4f7ad587a67af80b4f9f"} Oct 30 21:43:00.755569 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:00.755535 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:01.752559 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:01.752467 5801 generic.go:356] "Generic (PLEG): container finished" podID="968f2902-38b5-49f2-b219-89cf41af462d" containerID="31ec94472f02bd232bed28a224d01a997dc2edcc5ee0bbfea94533e23ec3783c" exitCode=0 Oct 30 21:43:01.755084 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:01.754231 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p" event={"ID":"968f2902-38b5-49f2-b219-89cf41af462d","Type":"ContainerDied","Data":"31ec94472f02bd232bed28a224d01a997dc2edcc5ee0bbfea94533e23ec3783c"} Oct 30 21:43:02.118518 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:02.083839 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b"] Oct 30 21:43:02.195235 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:02.195202 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b" Oct 30 21:43:02.304228 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:02.283100 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2d30a2-741b-46f7-a093-39fb4e9f6713-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b\" (UID: \"0f2d30a2-741b-46f7-a093-39fb4e9f6713\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b" Oct 30 21:43:02.304228 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:02.283189 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2d30a2-741b-46f7-a093-39fb4e9f6713-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b\" (UID: \"0f2d30a2-741b-46f7-a093-39fb4e9f6713\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b" Oct 30 21:43:02.304228 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:02.283327 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk2q4\" (UniqueName: \"kubernetes.io/projected/0f2d30a2-741b-46f7-a093-39fb4e9f6713-kube-api-access-zk2q4\") pod \"maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b\" (UID: \"0f2d30a2-741b-46f7-a093-39fb4e9f6713\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b" Oct 30 21:43:02.390832 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:02.389958 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b"] Oct 30 21:43:02.416789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:02.392253 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2d30a2-741b-46f7-a093-39fb4e9f6713-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b\" (UID: \"0f2d30a2-741b-46f7-a093-39fb4e9f6713\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b" Oct 30 21:43:02.416789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:02.392332 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2d30a2-741b-46f7-a093-39fb4e9f6713-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b\" (UID: \"0f2d30a2-741b-46f7-a093-39fb4e9f6713\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b" Oct 30 21:43:02.416789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:02.392468 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk2q4\" (UniqueName: \"kubernetes.io/projected/0f2d30a2-741b-46f7-a093-39fb4e9f6713-kube-api-access-zk2q4\") pod \"maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b\" (UID: \"0f2d30a2-741b-46f7-a093-39fb4e9f6713\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b" Oct 30 21:43:02.416789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:02.413217 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2d30a2-741b-46f7-a093-39fb4e9f6713-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b\" (UID: \"0f2d30a2-741b-46f7-a093-39fb4e9f6713\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b" Oct 30 21:43:02.555455 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:02.528176 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2d30a2-741b-46f7-a093-39fb4e9f6713-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b\" (UID: \"0f2d30a2-741b-46f7-a093-39fb4e9f6713\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b" Oct 30 21:43:02.608778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:02.585480 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:02.651361 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:02.645527 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk2q4\" (UniqueName: \"kubernetes.io/projected/0f2d30a2-741b-46f7-a093-39fb4e9f6713-kube-api-access-zk2q4\") pod \"maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b\" (UID: \"0f2d30a2-741b-46f7-a093-39fb4e9f6713\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b" Oct 30 21:43:02.752780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:02.737505 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:02.866257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:02.857253 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:02.890782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:02.875948 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:02.890782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:02.888770 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b" Oct 30 21:43:03.025819 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:03.024398 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded" Oct 30 21:43:03.169965 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:03.169905 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:03.228957 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:03.228850 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:03.250779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:03.242451 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:03.502827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:03.423728 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:03.502827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:03.446110 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:03.502827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:03.446669 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:43:03.503668 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:03.503194 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:03.546812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:03.545119 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:03.562794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:03.548289 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:43:03.671779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:03.668307 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:03.890776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:03.884792 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:03.890776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:03.885029 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:43:03.890776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:03.885484 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:43:04.070106 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:04.070066 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:04.170977 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:04.170874 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:43:04.293206 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:04.211028 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:43:04.294322 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:04.235924 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:04.294969 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:04.293498 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:43:04.346776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:04.340894 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:43:04.645835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:04.500666 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:04.645835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:04.500967 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:43:04.645835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:04.501222 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:43:04.645835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:04.501910 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:43:04.645835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:04.501970 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:43:04.645835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:04.502011 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:43:04.645835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:04.506060 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:04.645835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:04.512809 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:43:04.645835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:04.572642 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:04.645835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:04.613569 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:43:04.668709 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:04.667883 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:04.676775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:04.673332 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:43:04.709899 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:04.709858 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:04.857320 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:04.857265 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:43:04.884555 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:04.884496 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:04.885954 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:04.885922 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:04.945047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:04.929327 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:04.945047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:04.929763 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:43:05.074888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:05.074835 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:05.086776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:05.083188 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:43:05.239206 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:05.225535 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:05.315152 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:05.284007 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded" Oct 30 21:43:05.315152 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:05.293120 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:05.315152 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:05.294943 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:05.315152 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:05.296917 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:05.416515 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:05.416481 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:43:05.519228 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:05.519127 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:05.607811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:05.544981 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:05.607811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:05.545456 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:43:05.607811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:05.545687 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:43:05.608139 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:05.607884 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:43:05.700076 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:05.700023 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:05.778354 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:05.778252 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": context deadline exceeded" Oct 30 21:43:05.933920 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:05.890562 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:06.036084 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:06.035988 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:43:06.138781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:06.113492 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:06.944976 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:06.944927 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:06.951836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:06.281717 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:06.964778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:06.963782 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:43:06.965619 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:06.734858 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:06.995862 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:06.786968 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:06.996881 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:06.785913 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:43:07.001774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:06.787875 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:07.030779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:07.024001 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:43:07.139778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:07.097985 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:07.139778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:07.116980 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:43:07.139778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:07.123910 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:07.139778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:07.134030 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:43:07.237333 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:06.549872 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:43:07.237333 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:07.216925 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" podUID="db7b6603-3dfd-4f20-a362-da44b2fb9da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:07.334805 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:07.121837 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" podUID="b6307469-5f7d-41d9-9490-e8381ff9bbfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:07.532892 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:07.532733 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded" Oct 30 21:43:07.672779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:07.650004 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" podUID="edecde9f-7b29-4563-bf29-bcb748233470" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:07.820777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:07.807832 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:43:07.820777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:07.807920 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" podUID="b99efd6a-6f7d-444a-89dd-bf292349a7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:07.902584 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:07.878015 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p" Oct 30 21:43:07.958799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:07.948994 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:43:07.991932 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:07.984491 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:08.027818 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:43:08.018880 5801 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.819s" Oct 30 21:43:08.036778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.029821 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:43:08.068966 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.057077 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:43:08.135952 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.057144 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:43:08.145786 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.141715 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/968f2902-38b5-49f2-b219-89cf41af462d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/968f2902-38b5-49f2-b219-89cf41af462d-maas-api-openshift-ca\") pod \"968f2902-38b5-49f2-b219-89cf41af462d\" (UID: \"968f2902-38b5-49f2-b219-89cf41af462d\") " Oct 30 21:43:08.146789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.146376 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/968f2902-38b5-49f2-b219-89cf41af462d-kube-api-access-r8vp4\" (UniqueName: \"kubernetes.io/projected/968f2902-38b5-49f2-b219-89cf41af462d-kube-api-access-r8vp4\") pod \"968f2902-38b5-49f2-b219-89cf41af462d\" (UID: \"968f2902-38b5-49f2-b219-89cf41af462d\") " Oct 30 21:43:08.146789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.146497 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/968f2902-38b5-49f2-b219-89cf41af462d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/968f2902-38b5-49f2-b219-89cf41af462d-maas-api-xks-ca\") pod \"968f2902-38b5-49f2-b219-89cf41af462d\" (UID: \"968f2902-38b5-49f2-b219-89cf41af462d\") " Oct 30 21:43:08.314921 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.314878 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:43:08.315358 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.315329 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:43:08.315715 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.315693 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:43:08.340906 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.340862 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:43:08.342280 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.342248 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 21:43:08.343448 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.343423 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:43:08.361913 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.344534 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:43:08.363504 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.363452 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:43:08.364731 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.364689 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:43:08.486668 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.486626 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sk2kw" podUID="faea70fa-2fa1-45b8-9409-72fb5d589cc3" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.25:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:08.498410 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.498369 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/968f2902-38b5-49f2-b219-89cf41af462d-kube-api-access-r8vp4" pod "968f2902-38b5-49f2-b219-89cf41af462d" (UID: "968f2902-38b5-49f2-b219-89cf41af462d"). InnerVolumeSpecName "kube-api-access-r8vp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:08.533347 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.533210 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:08.571611 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.571579 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8vp4\" (UniqueName: \"kubernetes.io/projected/968f2902-38b5-49f2-b219-89cf41af462d-kube-api-access-r8vp4\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:08.601924 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.601881 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:08.602870 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.602837 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": context deadline exceeded" Oct 30 21:43:08.604020 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.603992 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/968f2902-38b5-49f2-b219-89cf41af462d-maas-api-openshift-ca" pod "968f2902-38b5-49f2-b219-89cf41af462d" (UID: "968f2902-38b5-49f2-b219-89cf41af462d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:08.634779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.631157 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p" Oct 30 21:43:08.688943 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.677797 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/968f2902-38b5-49f2-b219-89cf41af462d-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:08.725259 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.725155 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/968f2902-38b5-49f2-b219-89cf41af462d-maas-api-xks-ca" pod "968f2902-38b5-49f2-b219-89cf41af462d" (UID: "968f2902-38b5-49f2-b219-89cf41af462d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:08.780363 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.780298 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b" event={"ID":"0f2d30a2-741b-46f7-a093-39fb4e9f6713","Type":"ContainerStarted","Data":"cfd7155395a7916266851a8484898ce34ac31ae8ba91939e2e889ecd32da2cc1"} Oct 30 21:43:08.783989 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.783879 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p" event={"ID":"968f2902-38b5-49f2-b219-89cf41af462d","Type":"ContainerDied","Data":"c9ae5fbf762bfc0c024e975d3095c778b73aae8da6dc4f7ad587a67af80b4f9f"} Oct 30 21:43:08.783989 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.783939 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9ae5fbf762bfc0c024e975d3095c778b73aae8da6dc4f7ad587a67af80b4f9f" Oct 30 21:43:08.788550 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.788524 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/968f2902-38b5-49f2-b219-89cf41af462d-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:08.807543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.806825 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:08.950159 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:08.949169 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:09.067881 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:09.063838 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:43:09.071146 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:09.070277 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:09.124876 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:09.123780 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:09.173518 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:09.173246 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b"] Oct 30 21:43:09.304041 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:09.303785 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:09.334173 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:09.331654 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:09.334173 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:09.332064 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:43:09.334173 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:09.332332 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:43:09.356652 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:09.354189 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:09.356652 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:09.354581 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:43:09.356652 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:09.354885 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 21:43:09.364804 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:09.363883 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:09.364804 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:09.364231 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:43:09.376512 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:09.376471 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:09.571419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:09.571379 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:09.778354 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:09.778257 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:09.880692 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:09.880653 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:09.956356 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:09.956308 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:10.080538 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:10.079160 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:10.082896 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:10.082863 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:43:10.138941 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:10.138897 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:10.324778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:10.139848 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b" event={"ID":"0f2d30a2-741b-46f7-a093-39fb4e9f6713","Type":"ContainerStarted","Data":"5efd3820b797c651ae5a9107a42651fc8530aebb86603a66f3f148a2f40de5b4"} Oct 30 21:43:10.336977 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:10.336887 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:10.337657 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:10.337219 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:43:10.341771 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:10.337270 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:43:10.369063 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:10.369021 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:43:10.396805 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:10.378039 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:10.401531 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:10.395717 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:10.427826 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:10.395801 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:10.427826 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:10.395842 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:10.427826 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:10.395877 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:10.427826 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:10.395911 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:43:10.735830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:10.686139 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b" podStartSLOduration=8.686110327 podStartE2EDuration="8.686110327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:10.237770336 +0000 UTC m=+858.833587007" watchObservedRunningTime="2026-10-30 21:43:10.686110327 +0000 UTC m=+859.281926989" Oct 30 21:43:10.822879 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:10.805326 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:10.994940 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:10.992907 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:11.046993 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:11.046944 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:11.081158 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:11.081109 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:43:11.354657 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:11.354194 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:43:11.411927 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:11.402534 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:11.482673 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:11.482611 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6vklf" podUID="7e0434b3-6ab0-42c5-9c5b-e4592daadca7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:11.556174 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:11.494478 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:11.556174 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:11.494525 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 21:43:11.556174 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:11.494547 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:43:11.556174 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:11.494567 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:43:11.556174 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:11.494585 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:43:11.556174 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:11.516188 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx"] Oct 30 21:43:11.556174 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:11.526193 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:43:11.556174 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:11.526228 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:43:11.556174 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:11.526251 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:11.629848 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:11.629426 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:11.674302 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:11.674264 5801 generic.go:356] "Generic (PLEG): container finished" podID="0f2d30a2-741b-46f7-a093-39fb4e9f6713" containerID="5efd3820b797c651ae5a9107a42651fc8530aebb86603a66f3f148a2f40de5b4" exitCode=0 Oct 30 21:43:11.674672 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:11.674652 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b" event={"ID":"0f2d30a2-741b-46f7-a093-39fb4e9f6713","Type":"ContainerDied","Data":"5efd3820b797c651ae5a9107a42651fc8530aebb86603a66f3f148a2f40de5b4"} Oct 30 21:43:11.691276 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:11.676308 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" podUID="b6307469-5f7d-41d9-9490-e8381ff9bbfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:11.728115 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:11.727819 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" podUID="edecde9f-7b29-4563-bf29-bcb748233470" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:11.743634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:11.743599 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:11.841282 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:11.841231 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h" podUID="edecde9f-7b29-4563-bf29-bcb748233470" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:12.015053 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.014935 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:12.086478 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.086432 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:12.250621 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.250581 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:12.354912 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.354844 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:12.358614 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.358207 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:43:12.363471 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.360130 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:43:12.363471 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.362917 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:43:12.363737 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.363712 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:12.403940 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.403025 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:43:12.501784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.500212 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:43:12.530871 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.530834 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:43:12.533341 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.533311 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:12.533838 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.533739 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:12.534231 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.534208 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:12.534477 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.534395 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:12.539636 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.539618 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:43:12.597798 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.595446 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:43:12.597798 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.595492 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:43:12.659404 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.659328 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:43:12.660317 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.660297 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:43:12.661704 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.661679 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:43:12.663204 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.662893 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:43:12.664711 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.664132 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:43:12.677739 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.677721 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:43:12.910843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:12.910772 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b" Oct 30 21:43:13.038581 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:13.038159 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:13.049254 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:13.049230 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f2d30a2-741b-46f7-a093-39fb4e9f6713-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2d30a2-741b-46f7-a093-39fb4e9f6713-maas-api-openshift-ca\") pod \"0f2d30a2-741b-46f7-a093-39fb4e9f6713\" (UID: \"0f2d30a2-741b-46f7-a093-39fb4e9f6713\") " Oct 30 21:43:13.049392 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:13.049358 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f2d30a2-741b-46f7-a093-39fb4e9f6713-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2d30a2-741b-46f7-a093-39fb4e9f6713-maas-api-xks-ca\") pod \"0f2d30a2-741b-46f7-a093-39fb4e9f6713\" (UID: \"0f2d30a2-741b-46f7-a093-39fb4e9f6713\") " Oct 30 21:43:13.049467 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:13.049441 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f2d30a2-741b-46f7-a093-39fb4e9f6713-kube-api-access-zk2q4\" (UniqueName: \"kubernetes.io/projected/0f2d30a2-741b-46f7-a093-39fb4e9f6713-kube-api-access-zk2q4\") pod \"0f2d30a2-741b-46f7-a093-39fb4e9f6713\" (UID: \"0f2d30a2-741b-46f7-a093-39fb4e9f6713\") " Oct 30 21:43:13.049639 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:13.049613 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2d30a2-741b-46f7-a093-39fb4e9f6713-maas-api-openshift-ca" pod "0f2d30a2-741b-46f7-a093-39fb4e9f6713" (UID: "0f2d30a2-741b-46f7-a093-39fb4e9f6713"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:13.050035 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:13.050015 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2d30a2-741b-46f7-a093-39fb4e9f6713-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:13.051301 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:13.051274 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f2d30a2-741b-46f7-a093-39fb4e9f6713-maas-api-xks-ca" pod "0f2d30a2-741b-46f7-a093-39fb4e9f6713" (UID: "0f2d30a2-741b-46f7-a093-39fb4e9f6713"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:13.051458 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:13.051433 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2d30a2-741b-46f7-a093-39fb4e9f6713-kube-api-access-zk2q4" pod "0f2d30a2-741b-46f7-a093-39fb4e9f6713" (UID: "0f2d30a2-741b-46f7-a093-39fb4e9f6713"). InnerVolumeSpecName "kube-api-access-zk2q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:13.151418 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:13.151385 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2d30a2-741b-46f7-a093-39fb4e9f6713-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:13.151418 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:13.151417 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk2q4\" (UniqueName: \"kubernetes.io/projected/0f2d30a2-741b-46f7-a093-39fb4e9f6713-kube-api-access-zk2q4\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:13.707276 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:13.707248 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b" Oct 30 21:43:13.707642 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:13.707286 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b" event={"ID":"0f2d30a2-741b-46f7-a093-39fb4e9f6713","Type":"ContainerDied","Data":"cfd7155395a7916266851a8484898ce34ac31ae8ba91939e2e889ecd32da2cc1"} Oct 30 21:43:13.707642 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:13.707333 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfd7155395a7916266851a8484898ce34ac31ae8ba91939e2e889ecd32da2cc1" Oct 30 21:43:13.982204 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:13.982168 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:14.798046 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:14.798010 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:15.018253 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:15.018214 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:15.175187 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:15.175156 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p"] Oct 30 21:43:15.185463 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:15.185431 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-938e67d1-slf4p"] Oct 30 21:43:15.196503 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:15.196469 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:15.371140 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:15.369208 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:15.393741 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:15.393498 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79"] Oct 30 21:43:15.394454 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:15.394309 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" podUID="c2c309a8-0866-464c-be99-67d454090ea6" containerName="istio-proxy" containerID="cri-o://87df9029cf7c4d8172e4dfe8c2b274c864b8c83ebb78f0cc31442563b4b7df4c" gracePeriod=30 Oct 30 21:43:15.473462 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:15.473380 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:15.535441 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:15.535405 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="968f2902-38b5-49f2-b219-89cf41af462d" path="/var/lib/kubelet/pods/968f2902-38b5-49f2-b219-89cf41af462d/volumes" Oct 30 21:43:15.673782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:15.673484 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:15.824885 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:15.824835 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:16.112082 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:16.111998 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:16.466266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:16.466197 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:16.646417 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:16.646379 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:16.983725 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:16.983681 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:17.207889 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:17.207852 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx" podUID="516bdeb1-af0e-44c0-9187-20dd69d16099" containerName="payload-processing" containerID="cri-o://efe36aadc803f54e7706139ee3366f5f6eac6058fa800efa21743f3311b7c75d" gracePeriod=25 Oct 30 21:43:17.237972 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:17.237928 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:17.395832 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:17.392827 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:17.710620 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:17.710579 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx" Oct 30 21:43:17.828827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:17.828746 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/516bdeb1-af0e-44c0-9187-20dd69d16099-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/516bdeb1-af0e-44c0-9187-20dd69d16099-plugins-config-volume\") pod \"516bdeb1-af0e-44c0-9187-20dd69d16099\" (UID: \"516bdeb1-af0e-44c0-9187-20dd69d16099\") " Oct 30 21:43:17.829091 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:17.829065 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/516bdeb1-af0e-44c0-9187-20dd69d16099-kube-api-access-64qbc\" (UniqueName: \"kubernetes.io/projected/516bdeb1-af0e-44c0-9187-20dd69d16099-kube-api-access-64qbc\") pod \"516bdeb1-af0e-44c0-9187-20dd69d16099\" (UID: \"516bdeb1-af0e-44c0-9187-20dd69d16099\") " Oct 30 21:43:17.830236 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:17.830202 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/516bdeb1-af0e-44c0-9187-20dd69d16099-plugins-config-volume" pod "516bdeb1-af0e-44c0-9187-20dd69d16099" (UID: "516bdeb1-af0e-44c0-9187-20dd69d16099"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:17.834796 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:17.833909 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:17.834933 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:17.834902 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/516bdeb1-af0e-44c0-9187-20dd69d16099-kube-api-access-64qbc" pod "516bdeb1-af0e-44c0-9187-20dd69d16099" (UID: "516bdeb1-af0e-44c0-9187-20dd69d16099"). InnerVolumeSpecName "kube-api-access-64qbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:17.847477 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:17.847443 5801 generic.go:356] "Generic (PLEG): container finished" podID="516bdeb1-af0e-44c0-9187-20dd69d16099" containerID="efe36aadc803f54e7706139ee3366f5f6eac6058fa800efa21743f3311b7c75d" exitCode=0 Oct 30 21:43:17.847628 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:17.847515 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx" event={"ID":"516bdeb1-af0e-44c0-9187-20dd69d16099","Type":"ContainerDied","Data":"efe36aadc803f54e7706139ee3366f5f6eac6058fa800efa21743f3311b7c75d"} Oct 30 21:43:17.847628 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:17.847568 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx" event={"ID":"516bdeb1-af0e-44c0-9187-20dd69d16099","Type":"ContainerDied","Data":"7febf404bf0bb6d80dd27a85d187bd69ddc519929834a5bb8d7afd0c7f9d6eac"} Oct 30 21:43:17.847628 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:17.847595 5801 scope.go:122] "RemoveContainer" containerID="efe36aadc803f54e7706139ee3366f5f6eac6058fa800efa21743f3311b7c75d" Oct 30 21:43:17.847927 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:17.847786 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx" Oct 30 21:43:17.880689 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:17.880663 5801 scope.go:122] "RemoveContainer" containerID="efe36aadc803f54e7706139ee3366f5f6eac6058fa800efa21743f3311b7c75d" Oct 30 21:43:17.881314 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:43:17.881071 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efe36aadc803f54e7706139ee3366f5f6eac6058fa800efa21743f3311b7c75d\": container with ID starting with efe36aadc803f54e7706139ee3366f5f6eac6058fa800efa21743f3311b7c75d not found: ID does not exist" containerID="efe36aadc803f54e7706139ee3366f5f6eac6058fa800efa21743f3311b7c75d" Oct 30 21:43:17.881314 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:17.881114 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efe36aadc803f54e7706139ee3366f5f6eac6058fa800efa21743f3311b7c75d"} err="failed to get container status \"efe36aadc803f54e7706139ee3366f5f6eac6058fa800efa21743f3311b7c75d\": rpc error: code = NotFound desc = could not find container \"efe36aadc803f54e7706139ee3366f5f6eac6058fa800efa21743f3311b7c75d\": container with ID starting with efe36aadc803f54e7706139ee3366f5f6eac6058fa800efa21743f3311b7c75d not found: ID does not exist" Oct 30 21:43:17.914103 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:17.914070 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx"] Oct 30 21:43:17.920749 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:17.920724 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-938e67d1-cdf8fd67f-pd7gx"] Oct 30 21:43:17.931352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:17.931325 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/516bdeb1-af0e-44c0-9187-20dd69d16099-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:17.931352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:17.931350 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64qbc\" (UniqueName: \"kubernetes.io/projected/516bdeb1-af0e-44c0-9187-20dd69d16099-kube-api-access-64qbc\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:18.569859 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:18.569820 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:19.339035 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:19.338998 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:19.526635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:19.526604 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="516bdeb1-af0e-44c0-9187-20dd69d16099" path="/var/lib/kubelet/pods/516bdeb1-af0e-44c0-9187-20dd69d16099/volumes" Oct 30 21:43:19.560827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:19.560791 5801 patch_prober.go:29] interesting pod/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 30 21:43:19.561012 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:19.560846 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" podUID="c2c309a8-0866-464c-be99-67d454090ea6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:43:20.341581 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.341549 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:20.700550 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.700524 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:43:20.788910 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.788870 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-workload-certs\") pod \"c2c309a8-0866-464c-be99-67d454090ea6\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " Oct 30 21:43:20.789113 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.788926 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-credential-socket\") pod \"c2c309a8-0866-464c-be99-67d454090ea6\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " Oct 30 21:43:20.789113 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.788982 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-istio-data\") pod \"c2c309a8-0866-464c-be99-67d454090ea6\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " Oct 30 21:43:20.789113 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.789051 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-istio-envoy\") pod \"c2c309a8-0866-464c-be99-67d454090ea6\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " Oct 30 21:43:20.789113 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.789098 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c309a8-0866-464c-be99-67d454090ea6-kube-api-access-ts5vh\" (UniqueName: \"kubernetes.io/projected/c2c309a8-0866-464c-be99-67d454090ea6-kube-api-access-ts5vh\") pod \"c2c309a8-0866-464c-be99-67d454090ea6\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " Oct 30 21:43:20.789343 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.789154 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2c309a8-0866-464c-be99-67d454090ea6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c309a8-0866-464c-be99-67d454090ea6-istio-podinfo\") pod \"c2c309a8-0866-464c-be99-67d454090ea6\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " Oct 30 21:43:20.789343 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.789200 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c309a8-0866-464c-be99-67d454090ea6-istio-token\" (UniqueName: \"kubernetes.io/projected/c2c309a8-0866-464c-be99-67d454090ea6-istio-token\") pod \"c2c309a8-0866-464c-be99-67d454090ea6\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " Oct 30 21:43:20.789343 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.789194 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-workload-certs" pod "c2c309a8-0866-464c-be99-67d454090ea6" (UID: "c2c309a8-0866-464c-be99-67d454090ea6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:43:20.789343 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.789307 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2c309a8-0866-464c-be99-67d454090ea6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c309a8-0866-464c-be99-67d454090ea6-istiod-ca-cert\") pod \"c2c309a8-0866-464c-be99-67d454090ea6\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " Oct 30 21:43:20.789343 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.789335 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-workload-socket\") pod \"c2c309a8-0866-464c-be99-67d454090ea6\" (UID: \"c2c309a8-0866-464c-be99-67d454090ea6\") " Oct 30 21:43:20.790358 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.790334 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:20.790845 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.790821 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-workload-socket" pod "c2c309a8-0866-464c-be99-67d454090ea6" (UID: "c2c309a8-0866-464c-be99-67d454090ea6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:43:20.791139 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.791087 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-istio-data" pod "c2c309a8-0866-464c-be99-67d454090ea6" (UID: "c2c309a8-0866-464c-be99-67d454090ea6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:43:20.791518 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.791479 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2c309a8-0866-464c-be99-67d454090ea6-istiod-ca-cert" pod "c2c309a8-0866-464c-be99-67d454090ea6" (UID: "c2c309a8-0866-464c-be99-67d454090ea6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:20.792359 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.792177 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2c309a8-0866-464c-be99-67d454090ea6-istio-podinfo" pod "c2c309a8-0866-464c-be99-67d454090ea6" (UID: "c2c309a8-0866-464c-be99-67d454090ea6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:43:20.792359 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.792226 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c309a8-0866-464c-be99-67d454090ea6-kube-api-access-ts5vh" pod "c2c309a8-0866-464c-be99-67d454090ea6" (UID: "c2c309a8-0866-464c-be99-67d454090ea6"). InnerVolumeSpecName "kube-api-access-ts5vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:20.792359 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.792270 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-credential-socket" pod "c2c309a8-0866-464c-be99-67d454090ea6" (UID: "c2c309a8-0866-464c-be99-67d454090ea6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:43:20.793141 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.793104 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-istio-envoy" pod "c2c309a8-0866-464c-be99-67d454090ea6" (UID: "c2c309a8-0866-464c-be99-67d454090ea6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:43:20.793741 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.793708 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c309a8-0866-464c-be99-67d454090ea6-istio-token" pod "c2c309a8-0866-464c-be99-67d454090ea6" (UID: "c2c309a8-0866-464c-be99-67d454090ea6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:20.887008 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.886974 5801 generic.go:356] "Generic (PLEG): container finished" podID="c2c309a8-0866-464c-be99-67d454090ea6" containerID="87df9029cf7c4d8172e4dfe8c2b274c864b8c83ebb78f0cc31442563b4b7df4c" exitCode=0 Oct 30 21:43:20.887146 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.887020 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" Oct 30 21:43:20.887146 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.887052 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" event={"ID":"c2c309a8-0866-464c-be99-67d454090ea6","Type":"ContainerDied","Data":"87df9029cf7c4d8172e4dfe8c2b274c864b8c83ebb78f0cc31442563b4b7df4c"} Oct 30 21:43:20.887146 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.887103 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79" event={"ID":"c2c309a8-0866-464c-be99-67d454090ea6","Type":"ContainerDied","Data":"adcf86b79f183c6ebdea09c8ced257baedb375074cc64baa1c331e45274461cb"} Oct 30 21:43:20.887146 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.887124 5801 scope.go:122] "RemoveContainer" containerID="87df9029cf7c4d8172e4dfe8c2b274c864b8c83ebb78f0cc31442563b4b7df4c" Oct 30 21:43:20.891724 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.891700 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:20.891843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.891729 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:20.891843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.891745 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:20.891843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.891776 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts5vh\" (UniqueName: \"kubernetes.io/projected/c2c309a8-0866-464c-be99-67d454090ea6-kube-api-access-ts5vh\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:20.891843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.891791 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c309a8-0866-464c-be99-67d454090ea6-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:20.891843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.891804 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c309a8-0866-464c-be99-67d454090ea6-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:20.891843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.891816 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c309a8-0866-464c-be99-67d454090ea6-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:20.891843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.891830 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c309a8-0866-464c-be99-67d454090ea6-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:20.903105 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.902360 5801 scope.go:122] "RemoveContainer" containerID="87df9029cf7c4d8172e4dfe8c2b274c864b8c83ebb78f0cc31442563b4b7df4c" Oct 30 21:43:20.904353 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:43:20.904023 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87df9029cf7c4d8172e4dfe8c2b274c864b8c83ebb78f0cc31442563b4b7df4c\": container with ID starting with 87df9029cf7c4d8172e4dfe8c2b274c864b8c83ebb78f0cc31442563b4b7df4c not found: ID does not exist" containerID="87df9029cf7c4d8172e4dfe8c2b274c864b8c83ebb78f0cc31442563b4b7df4c" Oct 30 21:43:20.904353 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.904061 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87df9029cf7c4d8172e4dfe8c2b274c864b8c83ebb78f0cc31442563b4b7df4c"} err="failed to get container status \"87df9029cf7c4d8172e4dfe8c2b274c864b8c83ebb78f0cc31442563b4b7df4c\": rpc error: code = NotFound desc = could not find container \"87df9029cf7c4d8172e4dfe8c2b274c864b8c83ebb78f0cc31442563b4b7df4c\": container with ID starting with 87df9029cf7c4d8172e4dfe8c2b274c864b8c83ebb78f0cc31442563b4b7df4c not found: ID does not exist" Oct 30 21:43:20.916795 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.916768 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79"] Oct 30 21:43:20.925794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:20.925769 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-938e67d1-gw-openshift-default-6987c58cc7-9dp79"] Oct 30 21:43:21.148372 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:21.148288 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:21.253486 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:21.253447 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:21.527251 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:21.527174 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2c309a8-0866-464c-be99-67d454090ea6" path="/var/lib/kubelet/pods/c2c309a8-0866-464c-be99-67d454090ea6/volumes" Oct 30 21:43:21.538366 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:21.535635 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:22.195476 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:22.195444 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn"] Oct 30 21:43:22.220096 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:22.219732 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s"] Oct 30 21:43:22.279028 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:22.278986 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:22.425221 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:22.425172 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:22.614774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:22.608958 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:22.820579 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:22.820498 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:22.937682 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:22.937200 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:23.559179 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:23.559143 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:24.381497 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:24.381466 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:24.481947 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:24.481908 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:25.561192 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:25.561150 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:26.363507 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:26.363469 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:27.195962 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.195923 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn" podUID="3b377cbc-6859-4ae1-bde2-bac7680bb7d2" containerName="payload-processing" containerID="cri-o://13fccb87a2effea7bbe66e6253bf51b35a565118d7747ec3d92a275de215b187" gracePeriod=25 Oct 30 21:43:27.220556 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.220527 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s" podUID="6f0d6f19-bad5-415d-8230-125bf39ae64f" containerName="payload-pre-processing" containerID="cri-o://a2630c77f57453a2852a6b6be603af9b7e0e28a866502ed8aeec3acba24f7be5" gracePeriod=25 Oct 30 21:43:27.299044 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.299007 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:27.571082 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.571056 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn" Oct 30 21:43:27.620411 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.620391 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s" Oct 30 21:43:27.678000 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.677967 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b377cbc-6859-4ae1-bde2-bac7680bb7d2-kube-api-access-cklx9\" (UniqueName: \"kubernetes.io/projected/3b377cbc-6859-4ae1-bde2-bac7680bb7d2-kube-api-access-cklx9\") pod \"3b377cbc-6859-4ae1-bde2-bac7680bb7d2\" (UID: \"3b377cbc-6859-4ae1-bde2-bac7680bb7d2\") " Oct 30 21:43:27.678178 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.678081 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0d6f19-bad5-415d-8230-125bf39ae64f-kube-api-access-jqmpc\" (UniqueName: \"kubernetes.io/projected/6f0d6f19-bad5-415d-8230-125bf39ae64f-kube-api-access-jqmpc\") pod \"6f0d6f19-bad5-415d-8230-125bf39ae64f\" (UID: \"6f0d6f19-bad5-415d-8230-125bf39ae64f\") " Oct 30 21:43:27.678178 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.678128 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b377cbc-6859-4ae1-bde2-bac7680bb7d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b377cbc-6859-4ae1-bde2-bac7680bb7d2-plugins-config-volume\") pod \"3b377cbc-6859-4ae1-bde2-bac7680bb7d2\" (UID: \"3b377cbc-6859-4ae1-bde2-bac7680bb7d2\") " Oct 30 21:43:27.678341 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.678324 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f0d6f19-bad5-415d-8230-125bf39ae64f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0d6f19-bad5-415d-8230-125bf39ae64f-plugins-config-volume\") pod \"6f0d6f19-bad5-415d-8230-125bf39ae64f\" (UID: \"6f0d6f19-bad5-415d-8230-125bf39ae64f\") " Oct 30 21:43:27.678764 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.678711 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b377cbc-6859-4ae1-bde2-bac7680bb7d2-plugins-config-volume" pod "3b377cbc-6859-4ae1-bde2-bac7680bb7d2" (UID: "3b377cbc-6859-4ae1-bde2-bac7680bb7d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:27.678914 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.678885 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f0d6f19-bad5-415d-8230-125bf39ae64f-plugins-config-volume" pod "6f0d6f19-bad5-415d-8230-125bf39ae64f" (UID: "6f0d6f19-bad5-415d-8230-125bf39ae64f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:27.679211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.679183 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b377cbc-6859-4ae1-bde2-bac7680bb7d2-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:27.679306 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.679210 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0d6f19-bad5-415d-8230-125bf39ae64f-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:27.680527 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.680500 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0d6f19-bad5-415d-8230-125bf39ae64f-kube-api-access-jqmpc" pod "6f0d6f19-bad5-415d-8230-125bf39ae64f" (UID: "6f0d6f19-bad5-415d-8230-125bf39ae64f"). InnerVolumeSpecName "kube-api-access-jqmpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:27.681373 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.681352 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b377cbc-6859-4ae1-bde2-bac7680bb7d2-kube-api-access-cklx9" pod "3b377cbc-6859-4ae1-bde2-bac7680bb7d2" (UID: "3b377cbc-6859-4ae1-bde2-bac7680bb7d2"). InnerVolumeSpecName "kube-api-access-cklx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:27.781112 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.781043 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cklx9\" (UniqueName: \"kubernetes.io/projected/3b377cbc-6859-4ae1-bde2-bac7680bb7d2-kube-api-access-cklx9\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:27.781112 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.781076 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqmpc\" (UniqueName: \"kubernetes.io/projected/6f0d6f19-bad5-415d-8230-125bf39ae64f-kube-api-access-jqmpc\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:27.956932 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.956896 5801 generic.go:356] "Generic (PLEG): container finished" podID="6f0d6f19-bad5-415d-8230-125bf39ae64f" containerID="a2630c77f57453a2852a6b6be603af9b7e0e28a866502ed8aeec3acba24f7be5" exitCode=0 Oct 30 21:43:27.957114 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.956946 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s" Oct 30 21:43:27.957114 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.956976 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s" event={"ID":"6f0d6f19-bad5-415d-8230-125bf39ae64f","Type":"ContainerDied","Data":"a2630c77f57453a2852a6b6be603af9b7e0e28a866502ed8aeec3acba24f7be5"} Oct 30 21:43:27.957114 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.957042 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s" event={"ID":"6f0d6f19-bad5-415d-8230-125bf39ae64f","Type":"ContainerDied","Data":"4849012e02f4d5653f9f49e7f404e2ea9fbb4813ed10eca0980bef3b1fbaeb84"} Oct 30 21:43:27.957114 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.957071 5801 scope.go:122] "RemoveContainer" containerID="a2630c77f57453a2852a6b6be603af9b7e0e28a866502ed8aeec3acba24f7be5" Oct 30 21:43:27.958951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.958928 5801 generic.go:356] "Generic (PLEG): container finished" podID="3b377cbc-6859-4ae1-bde2-bac7680bb7d2" containerID="13fccb87a2effea7bbe66e6253bf51b35a565118d7747ec3d92a275de215b187" exitCode=0 Oct 30 21:43:27.959064 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.958965 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn" event={"ID":"3b377cbc-6859-4ae1-bde2-bac7680bb7d2","Type":"ContainerDied","Data":"13fccb87a2effea7bbe66e6253bf51b35a565118d7747ec3d92a275de215b187"} Oct 30 21:43:27.959064 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.959025 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn" event={"ID":"3b377cbc-6859-4ae1-bde2-bac7680bb7d2","Type":"ContainerDied","Data":"c85ad5a6b990b75413ed38c046dc0585ec9fff6a874a15781607e3ac3268d99b"} Oct 30 21:43:27.959064 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.958972 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn" Oct 30 21:43:27.972393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.972376 5801 scope.go:122] "RemoveContainer" containerID="a2630c77f57453a2852a6b6be603af9b7e0e28a866502ed8aeec3acba24f7be5" Oct 30 21:43:27.972784 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:43:27.972719 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2630c77f57453a2852a6b6be603af9b7e0e28a866502ed8aeec3acba24f7be5\": container with ID starting with a2630c77f57453a2852a6b6be603af9b7e0e28a866502ed8aeec3acba24f7be5 not found: ID does not exist" containerID="a2630c77f57453a2852a6b6be603af9b7e0e28a866502ed8aeec3acba24f7be5" Oct 30 21:43:27.972940 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.972907 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2630c77f57453a2852a6b6be603af9b7e0e28a866502ed8aeec3acba24f7be5"} err="failed to get container status \"a2630c77f57453a2852a6b6be603af9b7e0e28a866502ed8aeec3acba24f7be5\": rpc error: code = NotFound desc = could not find container \"a2630c77f57453a2852a6b6be603af9b7e0e28a866502ed8aeec3acba24f7be5\": container with ID starting with a2630c77f57453a2852a6b6be603af9b7e0e28a866502ed8aeec3acba24f7be5 not found: ID does not exist" Oct 30 21:43:27.972940 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.972940 5801 scope.go:122] "RemoveContainer" containerID="13fccb87a2effea7bbe66e6253bf51b35a565118d7747ec3d92a275de215b187" Oct 30 21:43:27.983295 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.983274 5801 scope.go:122] "RemoveContainer" containerID="13fccb87a2effea7bbe66e6253bf51b35a565118d7747ec3d92a275de215b187" Oct 30 21:43:27.983554 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:43:27.983534 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13fccb87a2effea7bbe66e6253bf51b35a565118d7747ec3d92a275de215b187\": container with ID starting with 13fccb87a2effea7bbe66e6253bf51b35a565118d7747ec3d92a275de215b187 not found: ID does not exist" containerID="13fccb87a2effea7bbe66e6253bf51b35a565118d7747ec3d92a275de215b187" Oct 30 21:43:27.983637 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.983564 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13fccb87a2effea7bbe66e6253bf51b35a565118d7747ec3d92a275de215b187"} err="failed to get container status \"13fccb87a2effea7bbe66e6253bf51b35a565118d7747ec3d92a275de215b187\": rpc error: code = NotFound desc = could not find container \"13fccb87a2effea7bbe66e6253bf51b35a565118d7747ec3d92a275de215b187\": container with ID starting with 13fccb87a2effea7bbe66e6253bf51b35a565118d7747ec3d92a275de215b187 not found: ID does not exist" Oct 30 21:43:27.994282 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:27.994259 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s"] Oct 30 21:43:28.000949 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:28.000831 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e19227f8-7cc886884c-sgr7s"] Oct 30 21:43:28.016257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:28.016171 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn"] Oct 30 21:43:28.021987 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:28.021965 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e19227f8-9fbb4875f-mtfxn"] Oct 30 21:43:28.043290 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:28.043230 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:28.843489 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:28.843458 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:29.525540 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:29.525507 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b377cbc-6859-4ae1-bde2-bac7680bb7d2" path="/var/lib/kubelet/pods/3b377cbc-6859-4ae1-bde2-bac7680bb7d2/volumes" Oct 30 21:43:29.525966 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:29.525950 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f0d6f19-bad5-415d-8230-125bf39ae64f" path="/var/lib/kubelet/pods/6f0d6f19-bad5-415d-8230-125bf39ae64f/volumes" Oct 30 21:43:29.664608 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:29.664572 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:29.785604 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:29.785515 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:30.290482 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:43:30.290444 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:43:30.291062 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:30.290973 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:30.401235 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:30.401191 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:30.535819 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:30.533593 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:30.831961 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:30.831923 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:30.876161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:30.876125 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b"] Oct 30 21:43:30.883305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:30.883277 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e19227f8-jsz9b"] Oct 30 21:43:31.045062 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:31.045007 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:31.158381 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:31.155909 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:31.569163 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:31.569130 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f2d30a2-741b-46f7-a093-39fb4e9f6713" path="/var/lib/kubelet/pods/0f2d30a2-741b-46f7-a093-39fb4e9f6713/volumes" Oct 30 21:43:31.718285 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:31.718249 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:32.061312 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:32.061279 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:32.211685 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:32.211650 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:32.526827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:32.524265 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:33.069520 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:33.069482 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:33.888657 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:33.888624 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:34.905300 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:34.905263 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:35.572730 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.572691 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n"] Oct 30 21:43:35.573423 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.573399 5801 cpu_manager.go:414] "containerMap: removing container" podUID="c2c309a8-0866-464c-be99-67d454090ea6" containerName="istio-proxy" Oct 30 21:43:35.573423 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.573422 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2c309a8-0866-464c-be99-67d454090ea6" containerName="istio-proxy" Oct 30 21:43:35.573613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.573442 5801 cpu_manager.go:414] "containerMap: removing container" podUID="0f2d30a2-741b-46f7-a093-39fb4e9f6713" containerName="revoke-keys" Oct 30 21:43:35.573613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.573449 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2d30a2-741b-46f7-a093-39fb4e9f6713" containerName="revoke-keys" Oct 30 21:43:35.573613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.573466 5801 cpu_manager.go:414] "containerMap: removing container" podUID="516bdeb1-af0e-44c0-9187-20dd69d16099" containerName="payload-processing" Oct 30 21:43:35.573613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.573473 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="516bdeb1-af0e-44c0-9187-20dd69d16099" containerName="payload-processing" Oct 30 21:43:35.573613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.573495 5801 cpu_manager.go:414] "containerMap: removing container" podUID="3b377cbc-6859-4ae1-bde2-bac7680bb7d2" containerName="payload-processing" Oct 30 21:43:35.573613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.573502 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b377cbc-6859-4ae1-bde2-bac7680bb7d2" containerName="payload-processing" Oct 30 21:43:35.573613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.573529 5801 cpu_manager.go:414] "containerMap: removing container" podUID="968f2902-38b5-49f2-b219-89cf41af462d" containerName="revoke-keys" Oct 30 21:43:35.573613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.573535 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="968f2902-38b5-49f2-b219-89cf41af462d" containerName="revoke-keys" Oct 30 21:43:35.573613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.573558 5801 cpu_manager.go:414] "containerMap: removing container" podUID="6f0d6f19-bad5-415d-8230-125bf39ae64f" containerName="payload-pre-processing" Oct 30 21:43:35.573613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.573566 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0d6f19-bad5-415d-8230-125bf39ae64f" containerName="payload-pre-processing" Oct 30 21:43:35.574116 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.573727 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2c309a8-0866-464c-be99-67d454090ea6" containerName="istio-proxy" Oct 30 21:43:35.574116 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.573742 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f2d30a2-741b-46f7-a093-39fb4e9f6713" containerName="revoke-keys" Oct 30 21:43:35.574116 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.573772 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="516bdeb1-af0e-44c0-9187-20dd69d16099" containerName="payload-processing" Oct 30 21:43:35.574116 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.573798 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b377cbc-6859-4ae1-bde2-bac7680bb7d2" containerName="payload-processing" Oct 30 21:43:35.574116 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.573809 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="968f2902-38b5-49f2-b219-89cf41af462d" containerName="revoke-keys" Oct 30 21:43:35.574116 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.573822 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0d6f19-bad5-415d-8230-125bf39ae64f" containerName="payload-pre-processing" Oct 30 21:43:35.580323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.580302 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n" Oct 30 21:43:35.583331 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.583307 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n"] Oct 30 21:43:35.584182 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.584160 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:43:35.585226 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.585205 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:43:35.585338 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.585241 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:43:35.585338 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.585262 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:43:35.729072 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.729041 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk476\" (UniqueName: \"kubernetes.io/projected/e363c71d-8d8f-4dfd-a9df-784687485af2-kube-api-access-sk476\") pod \"maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n\" (UID: \"e363c71d-8d8f-4dfd-a9df-784687485af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n" Oct 30 21:43:35.729252 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.729235 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e363c71d-8d8f-4dfd-a9df-784687485af2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n\" (UID: \"e363c71d-8d8f-4dfd-a9df-784687485af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n" Oct 30 21:43:35.729365 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.729348 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e363c71d-8d8f-4dfd-a9df-784687485af2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n\" (UID: \"e363c71d-8d8f-4dfd-a9df-784687485af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n" Oct 30 21:43:35.813267 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.813227 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:35.831400 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.831343 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk476\" (UniqueName: \"kubernetes.io/projected/e363c71d-8d8f-4dfd-a9df-784687485af2-kube-api-access-sk476\") pod \"maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n\" (UID: \"e363c71d-8d8f-4dfd-a9df-784687485af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n" Oct 30 21:43:35.831499 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.831491 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e363c71d-8d8f-4dfd-a9df-784687485af2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n\" (UID: \"e363c71d-8d8f-4dfd-a9df-784687485af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n" Oct 30 21:43:35.831584 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.831573 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e363c71d-8d8f-4dfd-a9df-784687485af2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n\" (UID: \"e363c71d-8d8f-4dfd-a9df-784687485af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n" Oct 30 21:43:35.832125 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.832101 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e363c71d-8d8f-4dfd-a9df-784687485af2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n\" (UID: \"e363c71d-8d8f-4dfd-a9df-784687485af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n" Oct 30 21:43:35.833882 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.833863 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e363c71d-8d8f-4dfd-a9df-784687485af2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n\" (UID: \"e363c71d-8d8f-4dfd-a9df-784687485af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n" Oct 30 21:43:35.853311 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.853282 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk476\" (UniqueName: \"kubernetes.io/projected/e363c71d-8d8f-4dfd-a9df-784687485af2-kube-api-access-sk476\") pod \"maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n\" (UID: \"e363c71d-8d8f-4dfd-a9df-784687485af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n" Oct 30 21:43:35.899103 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.899079 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n" Oct 30 21:43:35.956964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:35.956933 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:36.075778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:36.075731 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:36.206300 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:36.196952 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n"] Oct 30 21:43:37.023124 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.023021 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:37.087594 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.087555 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n" event={"ID":"e363c71d-8d8f-4dfd-a9df-784687485af2","Type":"ContainerStarted","Data":"638fe8a80408674f12e39639b98ffe18374d659bb6e499a226edfad31de8288c"} Oct 30 21:43:37.087823 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.087624 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n" event={"ID":"e363c71d-8d8f-4dfd-a9df-784687485af2","Type":"ContainerStarted","Data":"27898e3e69f8e5a13f4642c71d82cb15f7d9a1234a85c8b38f1e12d62f89c890"} Oct 30 21:43:37.196662 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.196595 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n" podStartSLOduration=2.196573405 podStartE2EDuration="2.196573405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:37.187336753 +0000 UTC m=+885.783153430" watchObservedRunningTime="2026-10-30 21:43:37.196573405 +0000 UTC m=+885.792390066" Oct 30 21:43:37.355004 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.354972 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:37.446036 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.445986 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6"] Oct 30 21:43:37.450665 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.450639 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6" Oct 30 21:43:37.459436 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.459409 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp"] Oct 30 21:43:37.460034 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.460007 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-484f7514\"" Oct 30 21:43:37.462692 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.462664 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-484f7514-dockercfg-mpz9p\"" Oct 30 21:43:37.465071 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.465050 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp" Oct 30 21:43:37.474894 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.471415 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6"] Oct 30 21:43:37.480729 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.480683 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp"] Oct 30 21:43:37.484382 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.484354 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:37.657317 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.657182 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c85995-b7c3-4a75-9c23-867f10fccb32-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6\" (UID: \"42c85995-b7c3-4a75-9c23-867f10fccb32\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6" Oct 30 21:43:37.657675 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.657599 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl5mk\" (UniqueName: \"kubernetes.io/projected/42c85995-b7c3-4a75-9c23-867f10fccb32-kube-api-access-rl5mk\") pod \"payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6\" (UID: \"42c85995-b7c3-4a75-9c23-867f10fccb32\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6" Oct 30 21:43:37.657908 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.657863 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a36cd401-1517-4440-b9a5-2a8bab48d14a-plugins-config-volume\") pod \"payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp\" (UID: \"a36cd401-1517-4440-b9a5-2a8bab48d14a\") " pod="openshift-ingress/payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp" Oct 30 21:43:37.658274 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.658236 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd7f8\" (UniqueName: \"kubernetes.io/projected/a36cd401-1517-4440-b9a5-2a8bab48d14a-kube-api-access-vd7f8\") pod \"payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp\" (UID: \"a36cd401-1517-4440-b9a5-2a8bab48d14a\") " pod="openshift-ingress/payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp" Oct 30 21:43:37.760806 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.760772 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c85995-b7c3-4a75-9c23-867f10fccb32-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6\" (UID: \"42c85995-b7c3-4a75-9c23-867f10fccb32\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6" Oct 30 21:43:37.761013 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.760840 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl5mk\" (UniqueName: \"kubernetes.io/projected/42c85995-b7c3-4a75-9c23-867f10fccb32-kube-api-access-rl5mk\") pod \"payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6\" (UID: \"42c85995-b7c3-4a75-9c23-867f10fccb32\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6" Oct 30 21:43:37.761013 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.760889 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a36cd401-1517-4440-b9a5-2a8bab48d14a-plugins-config-volume\") pod \"payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp\" (UID: \"a36cd401-1517-4440-b9a5-2a8bab48d14a\") " pod="openshift-ingress/payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp" Oct 30 21:43:37.761128 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.761052 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd7f8\" (UniqueName: \"kubernetes.io/projected/a36cd401-1517-4440-b9a5-2a8bab48d14a-kube-api-access-vd7f8\") pod \"payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp\" (UID: \"a36cd401-1517-4440-b9a5-2a8bab48d14a\") " pod="openshift-ingress/payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp" Oct 30 21:43:37.762334 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.762300 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c85995-b7c3-4a75-9c23-867f10fccb32-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6\" (UID: \"42c85995-b7c3-4a75-9c23-867f10fccb32\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6" Oct 30 21:43:37.763206 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.763182 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a36cd401-1517-4440-b9a5-2a8bab48d14a-plugins-config-volume\") pod \"payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp\" (UID: \"a36cd401-1517-4440-b9a5-2a8bab48d14a\") " pod="openshift-ingress/payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp" Oct 30 21:43:37.775069 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.775046 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd7f8\" (UniqueName: \"kubernetes.io/projected/a36cd401-1517-4440-b9a5-2a8bab48d14a-kube-api-access-vd7f8\") pod \"payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp\" (UID: \"a36cd401-1517-4440-b9a5-2a8bab48d14a\") " pod="openshift-ingress/payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp" Oct 30 21:43:37.779326 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.779309 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl5mk\" (UniqueName: \"kubernetes.io/projected/42c85995-b7c3-4a75-9c23-867f10fccb32-kube-api-access-rl5mk\") pod \"payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6\" (UID: \"42c85995-b7c3-4a75-9c23-867f10fccb32\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6" Oct 30 21:43:37.780184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.780139 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6" Oct 30 21:43:37.791963 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:37.791941 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp" Oct 30 21:43:38.099120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:38.099084 5801 generic.go:356] "Generic (PLEG): container finished" podID="e363c71d-8d8f-4dfd-a9df-784687485af2" containerID="638fe8a80408674f12e39639b98ffe18374d659bb6e499a226edfad31de8288c" exitCode=0 Oct 30 21:43:38.099489 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:38.099169 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n" event={"ID":"e363c71d-8d8f-4dfd-a9df-784687485af2","Type":"ContainerDied","Data":"638fe8a80408674f12e39639b98ffe18374d659bb6e499a226edfad31de8288c"} Oct 30 21:43:38.130288 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:43:38.129914 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36cd401_1517_4440_b9a5_2a8bab48d14a.slice/crio-2c3a1bf5c225b216db69ff9e6f5d2bb71a229ec40822ded1ca7c2839931643ad WatchSource:0}: Error finding container 2c3a1bf5c225b216db69ff9e6f5d2bb71a229ec40822ded1ca7c2839931643ad: Status 404 returned error can't find the container with id 2c3a1bf5c225b216db69ff9e6f5d2bb71a229ec40822ded1ca7c2839931643ad Oct 30 21:43:38.131609 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:38.131582 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp"] Oct 30 21:43:38.133462 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:43:38.133440 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c85995_b7c3_4a75_9c23_867f10fccb32.slice/crio-ccd92651840a0987948833deec62a604f68ddda65016da46208e791de1674bf4 WatchSource:0}: Error finding container ccd92651840a0987948833deec62a604f68ddda65016da46208e791de1674bf4: Status 404 returned error can't find the container with id ccd92651840a0987948833deec62a604f68ddda65016da46208e791de1674bf4 Oct 30 21:43:38.141415 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:38.141385 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6"] Oct 30 21:43:38.173840 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:38.173811 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:39.085900 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:39.085865 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:39.109542 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:39.109503 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp" event={"ID":"a36cd401-1517-4440-b9a5-2a8bab48d14a","Type":"ContainerStarted","Data":"9aa83d6cf7d5df33542ffebf9aac4deb28f9962445b50f56afb33ad1d478b0b4"} Oct 30 21:43:39.109962 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:39.109572 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp" event={"ID":"a36cd401-1517-4440-b9a5-2a8bab48d14a","Type":"ContainerStarted","Data":"2c3a1bf5c225b216db69ff9e6f5d2bb71a229ec40822ded1ca7c2839931643ad"} Oct 30 21:43:39.109962 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:39.109603 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp" Oct 30 21:43:39.112071 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:39.112041 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6" event={"ID":"42c85995-b7c3-4a75-9c23-867f10fccb32","Type":"ContainerStarted","Data":"dab09547aefc71ecb03628859f94d70a6cdc6fb33fcbf7092ff40245d7ab568e"} Oct 30 21:43:39.112691 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:39.112129 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6" event={"ID":"42c85995-b7c3-4a75-9c23-867f10fccb32","Type":"ContainerStarted","Data":"ccd92651840a0987948833deec62a604f68ddda65016da46208e791de1674bf4"} Oct 30 21:43:39.113015 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:39.112998 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6" Oct 30 21:43:39.150943 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:39.150876 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp" podStartSLOduration=2.1508555400000002 podStartE2EDuration="2.15085554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:39.148813772 +0000 UTC m=+887.744630432" watchObservedRunningTime="2026-10-30 21:43:39.15085554 +0000 UTC m=+887.746672201" Oct 30 21:43:39.175825 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:39.175766 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6" podStartSLOduration=2.175735351 podStartE2EDuration="2.175735351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:39.173885461 +0000 UTC m=+887.769702122" watchObservedRunningTime="2026-10-30 21:43:39.175735351 +0000 UTC m=+887.771552064" Oct 30 21:43:39.366364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:39.366329 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n" Oct 30 21:43:39.394054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:39.394021 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e363c71d-8d8f-4dfd-a9df-784687485af2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e363c71d-8d8f-4dfd-a9df-784687485af2-maas-api-openshift-ca\") pod \"e363c71d-8d8f-4dfd-a9df-784687485af2\" (UID: \"e363c71d-8d8f-4dfd-a9df-784687485af2\") " Oct 30 21:43:39.394235 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:39.394066 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e363c71d-8d8f-4dfd-a9df-784687485af2-kube-api-access-sk476\" (UniqueName: \"kubernetes.io/projected/e363c71d-8d8f-4dfd-a9df-784687485af2-kube-api-access-sk476\") pod \"e363c71d-8d8f-4dfd-a9df-784687485af2\" (UID: \"e363c71d-8d8f-4dfd-a9df-784687485af2\") " Oct 30 21:43:39.394235 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:39.394103 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e363c71d-8d8f-4dfd-a9df-784687485af2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e363c71d-8d8f-4dfd-a9df-784687485af2-maas-api-xks-ca\") pod \"e363c71d-8d8f-4dfd-a9df-784687485af2\" (UID: \"e363c71d-8d8f-4dfd-a9df-784687485af2\") " Oct 30 21:43:39.394460 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:39.394430 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e363c71d-8d8f-4dfd-a9df-784687485af2-maas-api-openshift-ca" pod "e363c71d-8d8f-4dfd-a9df-784687485af2" (UID: "e363c71d-8d8f-4dfd-a9df-784687485af2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:39.394964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:39.394936 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e363c71d-8d8f-4dfd-a9df-784687485af2-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:39.396461 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:39.396435 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e363c71d-8d8f-4dfd-a9df-784687485af2-kube-api-access-sk476" pod "e363c71d-8d8f-4dfd-a9df-784687485af2" (UID: "e363c71d-8d8f-4dfd-a9df-784687485af2"). InnerVolumeSpecName "kube-api-access-sk476". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:39.396975 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:39.396949 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e363c71d-8d8f-4dfd-a9df-784687485af2-maas-api-xks-ca" pod "e363c71d-8d8f-4dfd-a9df-784687485af2" (UID: "e363c71d-8d8f-4dfd-a9df-784687485af2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:39.496111 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:39.496082 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk476\" (UniqueName: \"kubernetes.io/projected/e363c71d-8d8f-4dfd-a9df-784687485af2-kube-api-access-sk476\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:39.496111 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:39.496110 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e363c71d-8d8f-4dfd-a9df-784687485af2-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:39.996630 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:39.996587 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:40.126386 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:40.126352 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n" Oct 30 21:43:40.126912 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:40.126366 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n" event={"ID":"e363c71d-8d8f-4dfd-a9df-784687485af2","Type":"ContainerDied","Data":"27898e3e69f8e5a13f4642c71d82cb15f7d9a1234a85c8b38f1e12d62f89c890"} Oct 30 21:43:40.126912 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:40.126445 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27898e3e69f8e5a13f4642c71d82cb15f7d9a1234a85c8b38f1e12d62f89c890" Oct 30 21:43:41.090785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:41.076093 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:41.923884 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:41.923843 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:42.671973 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:42.671931 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:43.442866 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:43.442827 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:44.176193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:44.176162 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:44.283994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:44.283957 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:44.406382 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:44.406346 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:45.044934 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:45.044905 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55412139-xqhwj"] Oct 30 21:43:45.051248 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:45.051220 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55412139-xqhwj"] Oct 30 21:43:45.277949 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:45.277918 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:45.526003 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:45.525969 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed020689-4336-4043-8e4a-dd496a48d080" path="/var/lib/kubelet/pods/ed020689-4336-4043-8e4a-dd496a48d080/volumes" Oct 30 21:43:45.835077 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:45.835046 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:45.948943 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:45.948909 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:46.186835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:46.185323 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:47.019309 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:47.019269 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:47.456462 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:47.456420 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d718596-66wfl"] Oct 30 21:43:47.457324 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:47.457302 5801 cpu_manager.go:414] "containerMap: removing container" podUID="e363c71d-8d8f-4dfd-a9df-784687485af2" containerName="revoke-keys" Oct 30 21:43:47.457429 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:47.457325 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e363c71d-8d8f-4dfd-a9df-784687485af2" containerName="revoke-keys" Oct 30 21:43:47.457568 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:47.457552 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="e363c71d-8d8f-4dfd-a9df-784687485af2" containerName="revoke-keys" Oct 30 21:43:47.461820 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:47.461798 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d718596-66wfl" Oct 30 21:43:47.468493 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:47.468472 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:43:47.468588 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:47.468507 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:43:47.468588 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:47.468541 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:43:47.469363 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:47.469342 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:43:47.478677 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:47.478649 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66d4a35-68fd-48c0-b12b-c4095304a75e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6d718596-66wfl\" (UID: \"f66d4a35-68fd-48c0-b12b-c4095304a75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d718596-66wfl" Oct 30 21:43:47.478925 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:47.478905 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66d4a35-68fd-48c0-b12b-c4095304a75e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6d718596-66wfl\" (UID: \"f66d4a35-68fd-48c0-b12b-c4095304a75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d718596-66wfl" Oct 30 21:43:47.479233 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:47.479214 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hr2g\" (UniqueName: \"kubernetes.io/projected/f66d4a35-68fd-48c0-b12b-c4095304a75e-kube-api-access-5hr2g\") pod \"maas-api-revoke-keys-e2e-ait-6d718596-66wfl\" (UID: \"f66d4a35-68fd-48c0-b12b-c4095304a75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d718596-66wfl" Oct 30 21:43:47.481123 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:47.481097 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d718596-66wfl"] Oct 30 21:43:47.581789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:47.581730 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66d4a35-68fd-48c0-b12b-c4095304a75e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6d718596-66wfl\" (UID: \"f66d4a35-68fd-48c0-b12b-c4095304a75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d718596-66wfl" Oct 30 21:43:47.581928 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:47.581842 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66d4a35-68fd-48c0-b12b-c4095304a75e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6d718596-66wfl\" (UID: \"f66d4a35-68fd-48c0-b12b-c4095304a75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d718596-66wfl" Oct 30 21:43:47.582114 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:47.582091 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hr2g\" (UniqueName: \"kubernetes.io/projected/f66d4a35-68fd-48c0-b12b-c4095304a75e-kube-api-access-5hr2g\") pod \"maas-api-revoke-keys-e2e-ait-6d718596-66wfl\" (UID: \"f66d4a35-68fd-48c0-b12b-c4095304a75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d718596-66wfl" Oct 30 21:43:47.582561 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:47.582534 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66d4a35-68fd-48c0-b12b-c4095304a75e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6d718596-66wfl\" (UID: \"f66d4a35-68fd-48c0-b12b-c4095304a75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d718596-66wfl" Oct 30 21:43:47.584231 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:47.584212 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66d4a35-68fd-48c0-b12b-c4095304a75e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6d718596-66wfl\" (UID: \"f66d4a35-68fd-48c0-b12b-c4095304a75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d718596-66wfl" Oct 30 21:43:47.595831 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:47.595809 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hr2g\" (UniqueName: \"kubernetes.io/projected/f66d4a35-68fd-48c0-b12b-c4095304a75e-kube-api-access-5hr2g\") pod \"maas-api-revoke-keys-e2e-ait-6d718596-66wfl\" (UID: \"f66d4a35-68fd-48c0-b12b-c4095304a75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d718596-66wfl" Oct 30 21:43:47.784605 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:47.784533 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d718596-66wfl" Oct 30 21:43:48.068432 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:48.068401 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:48.079683 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:43:48.079653 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66d4a35_68fd_48c0_b12b_c4095304a75e.slice/crio-04f003cb7fc47f757b682a338512b421145cfb400f7c437d8b089d44b01fe68b WatchSource:0}: Error finding container 04f003cb7fc47f757b682a338512b421145cfb400f7c437d8b089d44b01fe68b: Status 404 returned error can't find the container with id 04f003cb7fc47f757b682a338512b421145cfb400f7c437d8b089d44b01fe68b Oct 30 21:43:48.082325 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:48.082301 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d718596-66wfl"] Oct 30 21:43:48.270361 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:48.270327 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d718596-66wfl" event={"ID":"f66d4a35-68fd-48c0-b12b-c4095304a75e","Type":"ContainerStarted","Data":"04f003cb7fc47f757b682a338512b421145cfb400f7c437d8b089d44b01fe68b"} Oct 30 21:43:49.140512 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:49.140477 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:49.282298 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:49.282216 5801 generic.go:356] "Generic (PLEG): container finished" podID="f66d4a35-68fd-48c0-b12b-c4095304a75e" containerID="985eb575ae8aac117e9234cf0f862713e22b28434929f5e5cd9b558dcdbf4fdd" exitCode=0 Oct 30 21:43:49.282460 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:49.282304 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d718596-66wfl" event={"ID":"f66d4a35-68fd-48c0-b12b-c4095304a75e","Type":"ContainerDied","Data":"985eb575ae8aac117e9234cf0f862713e22b28434929f5e5cd9b558dcdbf4fdd"} Oct 30 21:43:50.141956 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:50.141927 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp" Oct 30 21:43:50.143908 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:50.143880 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6" Oct 30 21:43:50.173938 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:50.173885 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:51.283775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:51.283502 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:43:51.283775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:51.283643 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" podUID="db7b6603-3dfd-4f20-a362-da44b2fb9da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:51.342772 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:51.341847 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:51.499775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:51.498967 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:51.596950 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:51.587833 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:43:51.623780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:51.599147 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:43:51.701140 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:51.680427 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:43:51.877223 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:51.824603 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:43:51.877223 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:51.824948 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:43:51.877223 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:51.854711 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:52.023778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:51.998720 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:43:52.255829 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:52.243719 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:52.399658 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:52.399612 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:52.603897 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:52.310041 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:43:52.603897 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:52.295846 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:43:52.603897 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:52.464009 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:52.603897 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:52.531303 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:52.716469 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:52.716413 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:53.110834 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:43:53.052531 5801 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.495s" Oct 30 21:43:53.110834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:53.052615 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:43:53.110834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:53.052667 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:43:53.529779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:53.529305 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:53.971338 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:53.971283 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:53.979038 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:53.978996 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:43:54.088777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.085159 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:43:54.088777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.085566 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:43:54.088777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.085867 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:54.113832 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.089651 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:43:54.129969 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.129916 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:43:54.137788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.131061 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:43:54.137788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.131320 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:43:54.171952 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.138116 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded" Oct 30 21:43:54.171952 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.138258 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:54.171952 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.138670 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:54.171952 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.142031 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:54.171952 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.142293 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:54.235789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.235728 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:43:54.502530 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.502436 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d718596-66wfl" Oct 30 21:43:54.716130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.522884 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-515c34ba-crmc2"] Oct 30 21:43:54.716130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.580916 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d718596-66wfl" event={"ID":"f66d4a35-68fd-48c0-b12b-c4095304a75e","Type":"ContainerDied","Data":"04f003cb7fc47f757b682a338512b421145cfb400f7c437d8b089d44b01fe68b"} Oct 30 21:43:54.716130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.580956 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04f003cb7fc47f757b682a338512b421145cfb400f7c437d8b089d44b01fe68b" Oct 30 21:43:54.716130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.581005 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:43:54.716130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.581044 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:43:54.716130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.581081 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:43:54.716130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.581117 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:43:54.716130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.581150 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:43:54.716130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.581204 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:43:54.716130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.581239 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:43:54.716130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.581273 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:43:54.716130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.581319 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:43:54.716130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.581353 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:43:54.716130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.581387 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:43:54.716130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.551512 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:54.751621 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.751545 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-515c34ba-crmc2"] Oct 30 21:43:54.870692 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.870647 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:54.982944 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:54.982901 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:43:55.110505 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.107845 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:43:55.110505 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.108164 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:43:55.110505 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.108403 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:43:55.145264 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.140134 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:43:55.179796 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.151864 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:43:55.179796 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.174409 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f66d4a35-68fd-48c0-b12b-c4095304a75e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66d4a35-68fd-48c0-b12b-c4095304a75e-maas-api-openshift-ca\") pod \"f66d4a35-68fd-48c0-b12b-c4095304a75e\" (UID: \"f66d4a35-68fd-48c0-b12b-c4095304a75e\") " Oct 30 21:43:55.179796 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.174513 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f66d4a35-68fd-48c0-b12b-c4095304a75e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66d4a35-68fd-48c0-b12b-c4095304a75e-maas-api-xks-ca\") pod \"f66d4a35-68fd-48c0-b12b-c4095304a75e\" (UID: \"f66d4a35-68fd-48c0-b12b-c4095304a75e\") " Oct 30 21:43:55.179796 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.174793 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f66d4a35-68fd-48c0-b12b-c4095304a75e-kube-api-access-5hr2g\" (UniqueName: \"kubernetes.io/projected/f66d4a35-68fd-48c0-b12b-c4095304a75e-kube-api-access-5hr2g\") pod \"f66d4a35-68fd-48c0-b12b-c4095304a75e\" (UID: \"f66d4a35-68fd-48c0-b12b-c4095304a75e\") " Oct 30 21:43:55.180535 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.180484 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:43:55.181888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.181853 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:43:55.183101 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.183059 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:43:55.227238 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.183270 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:43:55.227689 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.183313 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:55.228081 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.183336 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:55.324670 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.283392 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded" Oct 30 21:43:55.324670 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.284986 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Oct 30 21:43:55.324670 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.289027 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66d4a35-68fd-48c0-b12b-c4095304a75e-kube-api-access-5hr2g" pod "f66d4a35-68fd-48c0-b12b-c4095304a75e" (UID: "f66d4a35-68fd-48c0-b12b-c4095304a75e"). InnerVolumeSpecName "kube-api-access-5hr2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:55.324670 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.316278 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:55.347351 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.327156 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f66d4a35-68fd-48c0-b12b-c4095304a75e-maas-api-xks-ca" pod "f66d4a35-68fd-48c0-b12b-c4095304a75e" (UID: "f66d4a35-68fd-48c0-b12b-c4095304a75e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:55.347863 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.347831 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f66d4a35-68fd-48c0-b12b-c4095304a75e-maas-api-openshift-ca" pod "f66d4a35-68fd-48c0-b12b-c4095304a75e" (UID: "f66d4a35-68fd-48c0-b12b-c4095304a75e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:55.515667 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.480318 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hr2g\" (UniqueName: \"kubernetes.io/projected/f66d4a35-68fd-48c0-b12b-c4095304a75e-kube-api-access-5hr2g\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:55.515667 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.480354 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66d4a35-68fd-48c0-b12b-c4095304a75e-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:55.515667 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.480373 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66d4a35-68fd-48c0-b12b-c4095304a75e-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:43:55.796299 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.774826 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": context deadline exceeded" Oct 30 21:43:55.833777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:55.818651 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d718596-66wfl" Oct 30 21:43:56.128776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.045605 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:56.128776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.119505 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:56.128776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.119984 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:56.128776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.120018 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:56.148824 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.148785 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:56.210959 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.210911 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:56.212247 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.212217 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:43:56.222875 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.222838 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:43:56.232777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.231638 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:56.264915 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.234710 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:43:56.267288 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.248920 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:56.322776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.321045 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:43:56.372774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.330404 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:56.430779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.401340 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc2deb4c-ba08-4812-9c4b-886b5c7a192d" path="/var/lib/kubelet/pods/fc2deb4c-ba08-4812-9c4b-886b5c7a192d/volumes" Oct 30 21:43:56.435131 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.435044 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:43:56.435131 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.435113 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:43:56.480534 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.480501 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:43:56.516800 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.516748 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:43:56.531794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.517711 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:43:56.540272 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.539896 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:43:56.557556 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.557524 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:43:56.626719 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.626677 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:43:56.671013 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.649569 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:43:56.671013 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.649644 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:43:56.671013 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.652602 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:43:56.671013 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.655933 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:43:56.671013 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.657157 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:43:56.688498 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.688414 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:56.946914 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:56.946832 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:57.079222 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:57.079187 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n"] Oct 30 21:43:57.085304 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:57.085272 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a5af9561-zpk6n"] Oct 30 21:43:57.093035 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:57.093003 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:57.250730 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:57.250638 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:57.375200 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:57.375164 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:57.533825 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:57.532923 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e363c71d-8d8f-4dfd-a9df-784687485af2" path="/var/lib/kubelet/pods/e363c71d-8d8f-4dfd-a9df-784687485af2/volumes" Oct 30 21:43:57.589982 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:57.589940 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:57.844804 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:57.843870 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:58.050140 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:58.050104 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:58.197301 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:58.195386 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:58.295929 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:58.295893 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:59.037395 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:59.037358 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e04ca9a6-p8zg4"] Oct 30 21:43:59.043359 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:59.043332 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e04ca9a6-p8zg4"] Oct 30 21:43:59.227440 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:59.227406 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:59.377291 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:59.377256 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:59.521785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:59.521714 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:59.527329 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:43:59.527304 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c751727a-5224-4b96-9a94-2430a78bd46a" path="/var/lib/kubelet/pods/c751727a-5224-4b96-9a94-2430a78bd46a/volumes" Oct 30 21:44:00.051344 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:00.051308 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e644c29-jb62q"] Oct 30 21:44:00.057411 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:00.057379 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c3de97-w4wjs"] Oct 30 21:44:00.063590 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:00.063565 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e644c29-jb62q"] Oct 30 21:44:00.069148 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:00.069126 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c3de97-w4wjs"] Oct 30 21:44:00.282590 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:44:00.282555 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:44:00.303367 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:00.296698 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:01.036585 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:01.036555 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea80e817-pg2sx"] Oct 30 21:44:01.043158 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:01.043127 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea80e817-pg2sx"] Oct 30 21:44:01.176656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:01.176617 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:01.545096 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:01.545051 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16e1bc47-ed65-41be-acf1-bfb2b33ab26f" path="/var/lib/kubelet/pods/16e1bc47-ed65-41be-acf1-bfb2b33ab26f/volumes" Oct 30 21:44:01.546746 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:01.546711 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56d5db44-d8e7-4c47-b9b0-1bc2d52c891b" path="/var/lib/kubelet/pods/56d5db44-d8e7-4c47-b9b0-1bc2d52c891b/volumes" Oct 30 21:44:01.547318 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:01.547295 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e" path="/var/lib/kubelet/pods/9d726c8c-7e7f-4b2b-ad3f-8c5ccdbdc59e/volumes" Oct 30 21:44:02.079796 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:02.079735 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:02.190935 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:02.190901 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-484f7514-zftzx"] Oct 30 21:44:02.191603 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:02.191589 5801 cpu_manager.go:414] "containerMap: removing container" podUID="f66d4a35-68fd-48c0-b12b-c4095304a75e" containerName="revoke-keys" Oct 30 21:44:02.191647 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:02.191616 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f66d4a35-68fd-48c0-b12b-c4095304a75e" containerName="revoke-keys" Oct 30 21:44:02.191728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:02.191719 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="f66d4a35-68fd-48c0-b12b-c4095304a75e" containerName="revoke-keys" Oct 30 21:44:02.197360 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:02.197338 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-484f7514-zftzx" Oct 30 21:44:02.205080 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:02.205055 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-484f7514-zftzx"] Oct 30 21:44:02.206290 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:02.206270 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:44:02.206479 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:02.206463 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:44:02.206604 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:02.206290 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:44:02.206707 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:02.206314 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:44:02.269910 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:02.269867 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-769b8\" (UniqueName: \"kubernetes.io/projected/09999956-3880-4ea3-b59b-edb4e2a1f757-kube-api-access-769b8\") pod \"maas-api-revoke-keys-e2e-ait-484f7514-zftzx\" (UID: \"09999956-3880-4ea3-b59b-edb4e2a1f757\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-484f7514-zftzx" Oct 30 21:44:02.270971 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:02.270262 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09999956-3880-4ea3-b59b-edb4e2a1f757-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-484f7514-zftzx\" (UID: \"09999956-3880-4ea3-b59b-edb4e2a1f757\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-484f7514-zftzx" Oct 30 21:44:02.270971 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:02.270356 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09999956-3880-4ea3-b59b-edb4e2a1f757-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-484f7514-zftzx\" (UID: \"09999956-3880-4ea3-b59b-edb4e2a1f757\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-484f7514-zftzx" Oct 30 21:44:02.372141 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:02.372069 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09999956-3880-4ea3-b59b-edb4e2a1f757-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-484f7514-zftzx\" (UID: \"09999956-3880-4ea3-b59b-edb4e2a1f757\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-484f7514-zftzx" Oct 30 21:44:02.372141 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:02.372123 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09999956-3880-4ea3-b59b-edb4e2a1f757-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-484f7514-zftzx\" (UID: \"09999956-3880-4ea3-b59b-edb4e2a1f757\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-484f7514-zftzx" Oct 30 21:44:02.372365 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:02.372160 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-769b8\" (UniqueName: \"kubernetes.io/projected/09999956-3880-4ea3-b59b-edb4e2a1f757-kube-api-access-769b8\") pod \"maas-api-revoke-keys-e2e-ait-484f7514-zftzx\" (UID: \"09999956-3880-4ea3-b59b-edb4e2a1f757\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-484f7514-zftzx" Oct 30 21:44:02.372808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:02.372748 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09999956-3880-4ea3-b59b-edb4e2a1f757-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-484f7514-zftzx\" (UID: \"09999956-3880-4ea3-b59b-edb4e2a1f757\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-484f7514-zftzx" Oct 30 21:44:02.374381 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:02.374357 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09999956-3880-4ea3-b59b-edb4e2a1f757-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-484f7514-zftzx\" (UID: \"09999956-3880-4ea3-b59b-edb4e2a1f757\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-484f7514-zftzx" Oct 30 21:44:02.384160 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:02.384135 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-769b8\" (UniqueName: \"kubernetes.io/projected/09999956-3880-4ea3-b59b-edb4e2a1f757-kube-api-access-769b8\") pod \"maas-api-revoke-keys-e2e-ait-484f7514-zftzx\" (UID: \"09999956-3880-4ea3-b59b-edb4e2a1f757\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-484f7514-zftzx" Oct 30 21:44:02.532456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:02.532425 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-484f7514-zftzx" Oct 30 21:44:02.826593 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:44:02.826564 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09999956_3880_4ea3_b59b_edb4e2a1f757.slice/crio-d914936b7dacecda81be07ac6b9d54180a17931dac66dea6605cec598e54fa03 WatchSource:0}: Error finding container d914936b7dacecda81be07ac6b9d54180a17931dac66dea6605cec598e54fa03: Status 404 returned error can't find the container with id d914936b7dacecda81be07ac6b9d54180a17931dac66dea6605cec598e54fa03 Oct 30 21:44:02.829333 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:02.829308 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-484f7514-zftzx"] Oct 30 21:44:02.845449 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:02.845420 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:02.908561 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:02.908531 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-484f7514-zftzx" event={"ID":"09999956-3880-4ea3-b59b-edb4e2a1f757","Type":"ContainerStarted","Data":"d914936b7dacecda81be07ac6b9d54180a17931dac66dea6605cec598e54fa03"} Oct 30 21:44:03.094822 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:03.094459 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:03.292912 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:03.292066 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:03.297421 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:03.297395 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d718596-66wfl"] Oct 30 21:44:03.305137 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:03.304433 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d718596-66wfl"] Oct 30 21:44:03.400623 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:03.400585 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:03.527491 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:03.527455 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f66d4a35-68fd-48c0-b12b-c4095304a75e" path="/var/lib/kubelet/pods/f66d4a35-68fd-48c0-b12b-c4095304a75e/volumes" Oct 30 21:44:03.921347 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:03.921306 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-484f7514-zftzx" event={"ID":"09999956-3880-4ea3-b59b-edb4e2a1f757","Type":"ContainerStarted","Data":"c85347b278842a802f33182da10d8fa1810acca660fde1dae0f0020c40002795"} Oct 30 21:44:03.955158 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:03.955107 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-484f7514-zftzx" podStartSLOduration=1.955092568 podStartE2EDuration="1.955092568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:03.947931522 +0000 UTC m=+912.543748201" watchObservedRunningTime="2026-10-30 21:44:03.955092568 +0000 UTC m=+912.550909227" Oct 30 21:44:04.109081 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:04.109034 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:04.570571 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:04.570530 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:04.932546 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:04.932468 5801 generic.go:356] "Generic (PLEG): container finished" podID="09999956-3880-4ea3-b59b-edb4e2a1f757" containerID="c85347b278842a802f33182da10d8fa1810acca660fde1dae0f0020c40002795" exitCode=0 Oct 30 21:44:04.932940 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:04.932561 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-484f7514-zftzx" event={"ID":"09999956-3880-4ea3-b59b-edb4e2a1f757","Type":"ContainerDied","Data":"c85347b278842a802f33182da10d8fa1810acca660fde1dae0f0020c40002795"} Oct 30 21:44:05.087682 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:05.087640 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:05.532844 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:05.532807 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:05.727779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:05.709350 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:06.143340 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:06.143310 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-484f7514-zftzx" Oct 30 21:44:06.256732 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:06.256654 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09999956-3880-4ea3-b59b-edb4e2a1f757-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09999956-3880-4ea3-b59b-edb4e2a1f757-maas-api-xks-ca\") pod \"09999956-3880-4ea3-b59b-edb4e2a1f757\" (UID: \"09999956-3880-4ea3-b59b-edb4e2a1f757\") " Oct 30 21:44:06.256732 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:06.256698 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09999956-3880-4ea3-b59b-edb4e2a1f757-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09999956-3880-4ea3-b59b-edb4e2a1f757-maas-api-openshift-ca\") pod \"09999956-3880-4ea3-b59b-edb4e2a1f757\" (UID: \"09999956-3880-4ea3-b59b-edb4e2a1f757\") " Oct 30 21:44:06.256938 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:06.256896 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09999956-3880-4ea3-b59b-edb4e2a1f757-kube-api-access-769b8\" (UniqueName: \"kubernetes.io/projected/09999956-3880-4ea3-b59b-edb4e2a1f757-kube-api-access-769b8\") pod \"09999956-3880-4ea3-b59b-edb4e2a1f757\" (UID: \"09999956-3880-4ea3-b59b-edb4e2a1f757\") " Oct 30 21:44:06.257105 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:06.257069 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09999956-3880-4ea3-b59b-edb4e2a1f757-maas-api-openshift-ca" pod "09999956-3880-4ea3-b59b-edb4e2a1f757" (UID: "09999956-3880-4ea3-b59b-edb4e2a1f757"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:06.257513 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:06.257488 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09999956-3880-4ea3-b59b-edb4e2a1f757-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:06.258806 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:06.258783 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09999956-3880-4ea3-b59b-edb4e2a1f757-maas-api-xks-ca" pod "09999956-3880-4ea3-b59b-edb4e2a1f757" (UID: "09999956-3880-4ea3-b59b-edb4e2a1f757"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:06.259024 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:06.259007 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09999956-3880-4ea3-b59b-edb4e2a1f757-kube-api-access-769b8" pod "09999956-3880-4ea3-b59b-edb4e2a1f757" (UID: "09999956-3880-4ea3-b59b-edb4e2a1f757"). InnerVolumeSpecName "kube-api-access-769b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:06.359021 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:06.358983 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-769b8\" (UniqueName: \"kubernetes.io/projected/09999956-3880-4ea3-b59b-edb4e2a1f757-kube-api-access-769b8\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:06.359021 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:06.359015 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09999956-3880-4ea3-b59b-edb4e2a1f757-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:06.662850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:06.662813 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:06.826165 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:06.826120 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:06.953656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:06.953580 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-484f7514-zftzx" Oct 30 21:44:06.953656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:06.953615 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-484f7514-zftzx" event={"ID":"09999956-3880-4ea3-b59b-edb4e2a1f757","Type":"ContainerDied","Data":"d914936b7dacecda81be07ac6b9d54180a17931dac66dea6605cec598e54fa03"} Oct 30 21:44:06.953889 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:06.953665 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d914936b7dacecda81be07ac6b9d54180a17931dac66dea6605cec598e54fa03" Oct 30 21:44:06.955485 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:06.955456 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:07.683122 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:07.683081 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:08.031607 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:08.031520 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-762d372d-z4w6z"] Oct 30 21:44:08.037374 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:08.037346 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-762d372d-z4w6z"] Oct 30 21:44:08.563550 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:08.563511 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:09.029635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:09.029545 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b2286e6-5v87x"] Oct 30 21:44:09.035693 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:09.035661 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b2286e6-5v87x"] Oct 30 21:44:09.526309 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:09.526273 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5899e00-98f2-4184-a863-4cafff2ae618" path="/var/lib/kubelet/pods/a5899e00-98f2-4184-a863-4cafff2ae618/volumes" Oct 30 21:44:09.526837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:09.526822 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c26fc0e5-bd68-4d01-9fb8-9c16c1971b74" path="/var/lib/kubelet/pods/c26fc0e5-bd68-4d01-9fb8-9c16c1971b74/volumes" Oct 30 21:44:09.533370 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:09.533337 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:10.010889 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:10.010850 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:10.080814 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:10.074680 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed7d1d1-9xnnw"] Oct 30 21:44:10.143775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:10.128623 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed7d1d1-9xnnw"] Oct 30 21:44:10.171804 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:10.171767 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:10.856606 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:10.856570 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:11.102554 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:11.102485 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8991907d-wxl9z"] Oct 30 21:44:11.112948 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:11.112861 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8991907d-wxl9z"] Oct 30 21:44:11.343848 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:11.343801 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:11.454382 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:11.454310 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt"] Oct 30 21:44:11.455475 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:11.455445 5801 cpu_manager.go:414] "containerMap: removing container" podUID="09999956-3880-4ea3-b59b-edb4e2a1f757" containerName="revoke-keys" Oct 30 21:44:11.455475 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:11.455473 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09999956-3880-4ea3-b59b-edb4e2a1f757" containerName="revoke-keys" Oct 30 21:44:11.455677 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:11.455662 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="09999956-3880-4ea3-b59b-edb4e2a1f757" containerName="revoke-keys" Oct 30 21:44:11.472774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:11.472714 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:11.481205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:11.481178 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt" Oct 30 21:44:11.491618 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:11.491086 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-1f725d-dockercfg-fh48c\"" Oct 30 21:44:11.498739 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:11.498710 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt"] Oct 30 21:44:11.534819 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:11.534779 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d067f135-fcc0-4578-bb69-6fb5a6ecf49e" path="/var/lib/kubelet/pods/d067f135-fcc0-4578-bb69-6fb5a6ecf49e/volumes" Oct 30 21:44:11.535506 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:11.535488 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f" path="/var/lib/kubelet/pods/f6dd41de-7a1e-4d5b-ae2c-0500b7ad116f/volumes" Oct 30 21:44:11.540947 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:11.540918 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-1f725d\"" Oct 30 21:44:11.602778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:11.595855 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95cj2\" (UniqueName: \"kubernetes.io/projected/f1db92e5-5f7e-42ee-b389-ac3690448d17-kube-api-access-95cj2\") pod \"payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt\" (UID: \"f1db92e5-5f7e-42ee-b389-ac3690448d17\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt" Oct 30 21:44:11.602778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:11.596129 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1db92e5-5f7e-42ee-b389-ac3690448d17-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt\" (UID: \"f1db92e5-5f7e-42ee-b389-ac3690448d17\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt" Oct 30 21:44:11.707444 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:11.707344 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95cj2\" (UniqueName: \"kubernetes.io/projected/f1db92e5-5f7e-42ee-b389-ac3690448d17-kube-api-access-95cj2\") pod \"payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt\" (UID: \"f1db92e5-5f7e-42ee-b389-ac3690448d17\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt" Oct 30 21:44:11.708781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:11.708741 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1db92e5-5f7e-42ee-b389-ac3690448d17-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt\" (UID: \"f1db92e5-5f7e-42ee-b389-ac3690448d17\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt" Oct 30 21:44:11.720469 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:11.720430 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1db92e5-5f7e-42ee-b389-ac3690448d17-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt\" (UID: \"f1db92e5-5f7e-42ee-b389-ac3690448d17\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt" Oct 30 21:44:11.745832 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:11.727550 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95cj2\" (UniqueName: \"kubernetes.io/projected/f1db92e5-5f7e-42ee-b389-ac3690448d17-kube-api-access-95cj2\") pod \"payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt\" (UID: \"f1db92e5-5f7e-42ee-b389-ac3690448d17\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt" Oct 30 21:44:11.805646 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:11.805606 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt" Oct 30 21:44:12.047383 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:12.047305 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06e3ae2e-pp6k5"] Oct 30 21:44:12.053398 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:12.053371 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06e3ae2e-pp6k5"] Oct 30 21:44:12.058699 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:12.058678 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e44bde8-l5x9w"] Oct 30 21:44:12.063824 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:12.063795 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e44bde8-l5x9w"] Oct 30 21:44:12.064678 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:12.064651 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:12.108119 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:12.106541 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt"] Oct 30 21:44:12.313617 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:12.313582 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp"] Oct 30 21:44:12.354406 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:12.354371 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6"] Oct 30 21:44:12.415930 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:12.415896 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:12.523802 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:12.523733 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:13.136023 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:13.135988 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt" event={"ID":"f1db92e5-5f7e-42ee-b389-ac3690448d17","Type":"ContainerStarted","Data":"93cd41f9752099fe0030a4393836e638a20b73e7bd170b5f88bcb7af4f200ba6"} Oct 30 21:44:13.136441 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:13.136049 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt" event={"ID":"f1db92e5-5f7e-42ee-b389-ac3690448d17","Type":"ContainerStarted","Data":"0d0e622ee5fab1b9b8120d136135acfc7a6a3317cc5cb0e002ad4c204787f4c2"} Oct 30 21:44:13.136441 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:13.136079 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt" Oct 30 21:44:13.184367 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:13.184315 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt" podStartSLOduration=2.18429939 podStartE2EDuration="2.18429939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:13.179473691 +0000 UTC m=+921.775290370" watchObservedRunningTime="2026-10-30 21:44:13.18429939 +0000 UTC m=+921.780116051" Oct 30 21:44:13.238955 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:13.238917 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:13.525525 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:13.525446 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44c9fa99-b9a5-452c-b352-481a0cf4ec4a" path="/var/lib/kubelet/pods/44c9fa99-b9a5-452c-b352-481a0cf4ec4a/volumes" Oct 30 21:44:13.526034 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:13.526017 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d07a906-016e-49df-88a4-3d3e3bf2dd32" path="/var/lib/kubelet/pods/5d07a906-016e-49df-88a4-3d3e3bf2dd32/volumes" Oct 30 21:44:14.038673 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:14.038632 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ae7f502-ppw8w"] Oct 30 21:44:14.044965 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:14.044937 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ae7f502-ppw8w"] Oct 30 21:44:14.162728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:14.162693 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:14.262577 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:14.262544 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:15.063165 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:15.063084 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:15.526914 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:15.526840 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2191dc8-cb0d-4967-a182-32d50b9ad13e" path="/var/lib/kubelet/pods/b2191dc8-cb0d-4967-a182-32d50b9ad13e/volumes" Oct 30 21:44:16.189200 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:16.189171 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:16.827121 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:16.827080 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:16.982878 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:16.982841 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:17.008246 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:17.008215 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-484f7514-zftzx"] Oct 30 21:44:17.013645 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:17.013619 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-484f7514-zftzx"] Oct 30 21:44:17.134437 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:17.134357 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:17.260786 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:17.254216 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:17.329244 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:17.326204 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp" podUID="a36cd401-1517-4440-b9a5-2a8bab48d14a" containerName="payload-processing" containerID="cri-o://9aa83d6cf7d5df33542ffebf9aac4deb28f9962445b50f56afb33ad1d478b0b4" gracePeriod=25 Oct 30 21:44:17.358169 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:17.356990 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6" podUID="42c85995-b7c3-4a75-9c23-867f10fccb32" containerName="payload-pre-processing" containerID="cri-o://dab09547aefc71ecb03628859f94d70a6cdc6fb33fcbf7092ff40245d7ab568e" gracePeriod=25 Oct 30 21:44:17.531588 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:17.531194 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09999956-3880-4ea3-b59b-edb4e2a1f757" path="/var/lib/kubelet/pods/09999956-3880-4ea3-b59b-edb4e2a1f757/volumes" Oct 30 21:44:17.720076 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:17.720044 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp" Oct 30 21:44:17.787375 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:17.787345 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6" Oct 30 21:44:17.880474 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:17.880446 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42c85995-b7c3-4a75-9c23-867f10fccb32-kube-api-access-rl5mk\" (UniqueName: \"kubernetes.io/projected/42c85995-b7c3-4a75-9c23-867f10fccb32-kube-api-access-rl5mk\") pod \"42c85995-b7c3-4a75-9c23-867f10fccb32\" (UID: \"42c85995-b7c3-4a75-9c23-867f10fccb32\") " Oct 30 21:44:17.880965 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:17.880552 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42c85995-b7c3-4a75-9c23-867f10fccb32-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c85995-b7c3-4a75-9c23-867f10fccb32-plugins-config-volume\") pod \"42c85995-b7c3-4a75-9c23-867f10fccb32\" (UID: \"42c85995-b7c3-4a75-9c23-867f10fccb32\") " Oct 30 21:44:17.880965 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:17.880722 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a36cd401-1517-4440-b9a5-2a8bab48d14a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a36cd401-1517-4440-b9a5-2a8bab48d14a-plugins-config-volume\") pod \"a36cd401-1517-4440-b9a5-2a8bab48d14a\" (UID: \"a36cd401-1517-4440-b9a5-2a8bab48d14a\") " Oct 30 21:44:17.880965 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:17.880825 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a36cd401-1517-4440-b9a5-2a8bab48d14a-kube-api-access-vd7f8\" (UniqueName: \"kubernetes.io/projected/a36cd401-1517-4440-b9a5-2a8bab48d14a-kube-api-access-vd7f8\") pod \"a36cd401-1517-4440-b9a5-2a8bab48d14a\" (UID: \"a36cd401-1517-4440-b9a5-2a8bab48d14a\") " Oct 30 21:44:17.881151 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:17.881016 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42c85995-b7c3-4a75-9c23-867f10fccb32-plugins-config-volume" pod "42c85995-b7c3-4a75-9c23-867f10fccb32" (UID: "42c85995-b7c3-4a75-9c23-867f10fccb32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:17.881151 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:17.881128 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a36cd401-1517-4440-b9a5-2a8bab48d14a-plugins-config-volume" pod "a36cd401-1517-4440-b9a5-2a8bab48d14a" (UID: "a36cd401-1517-4440-b9a5-2a8bab48d14a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:17.881428 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:17.881408 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a36cd401-1517-4440-b9a5-2a8bab48d14a-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:17.881428 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:17.881428 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c85995-b7c3-4a75-9c23-867f10fccb32-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:17.882744 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:17.882718 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c85995-b7c3-4a75-9c23-867f10fccb32-kube-api-access-rl5mk" pod "42c85995-b7c3-4a75-9c23-867f10fccb32" (UID: "42c85995-b7c3-4a75-9c23-867f10fccb32"). InnerVolumeSpecName "kube-api-access-rl5mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:17.883061 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:17.883041 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a36cd401-1517-4440-b9a5-2a8bab48d14a-kube-api-access-vd7f8" pod "a36cd401-1517-4440-b9a5-2a8bab48d14a" (UID: "a36cd401-1517-4440-b9a5-2a8bab48d14a"). InnerVolumeSpecName "kube-api-access-vd7f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:17.983059 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:17.982966 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl5mk\" (UniqueName: \"kubernetes.io/projected/42c85995-b7c3-4a75-9c23-867f10fccb32-kube-api-access-rl5mk\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:17.983059 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:17.983000 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd7f8\" (UniqueName: \"kubernetes.io/projected/a36cd401-1517-4440-b9a5-2a8bab48d14a-kube-api-access-vd7f8\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:18.062236 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:18.062197 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:18.190563 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:18.190528 5801 generic.go:356] "Generic (PLEG): container finished" podID="a36cd401-1517-4440-b9a5-2a8bab48d14a" containerID="9aa83d6cf7d5df33542ffebf9aac4deb28f9962445b50f56afb33ad1d478b0b4" exitCode=0 Oct 30 21:44:18.190738 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:18.190576 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp" Oct 30 21:44:18.190738 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:18.190608 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp" event={"ID":"a36cd401-1517-4440-b9a5-2a8bab48d14a","Type":"ContainerDied","Data":"9aa83d6cf7d5df33542ffebf9aac4deb28f9962445b50f56afb33ad1d478b0b4"} Oct 30 21:44:18.190738 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:18.190668 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp" event={"ID":"a36cd401-1517-4440-b9a5-2a8bab48d14a","Type":"ContainerDied","Data":"2c3a1bf5c225b216db69ff9e6f5d2bb71a229ec40822ded1ca7c2839931643ad"} Oct 30 21:44:18.190738 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:18.190698 5801 scope.go:122] "RemoveContainer" containerID="9aa83d6cf7d5df33542ffebf9aac4deb28f9962445b50f56afb33ad1d478b0b4" Oct 30 21:44:18.192335 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:18.192315 5801 generic.go:356] "Generic (PLEG): container finished" podID="42c85995-b7c3-4a75-9c23-867f10fccb32" containerID="dab09547aefc71ecb03628859f94d70a6cdc6fb33fcbf7092ff40245d7ab568e" exitCode=0 Oct 30 21:44:18.192428 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:18.192358 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6" event={"ID":"42c85995-b7c3-4a75-9c23-867f10fccb32","Type":"ContainerDied","Data":"dab09547aefc71ecb03628859f94d70a6cdc6fb33fcbf7092ff40245d7ab568e"} Oct 30 21:44:18.192428 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:18.192375 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6" Oct 30 21:44:18.192428 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:18.192414 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6" event={"ID":"42c85995-b7c3-4a75-9c23-867f10fccb32","Type":"ContainerDied","Data":"ccd92651840a0987948833deec62a604f68ddda65016da46208e791de1674bf4"} Oct 30 21:44:18.220169 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:18.220150 5801 scope.go:122] "RemoveContainer" containerID="9aa83d6cf7d5df33542ffebf9aac4deb28f9962445b50f56afb33ad1d478b0b4" Oct 30 21:44:18.220441 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:44:18.220424 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aa83d6cf7d5df33542ffebf9aac4deb28f9962445b50f56afb33ad1d478b0b4\": container with ID starting with 9aa83d6cf7d5df33542ffebf9aac4deb28f9962445b50f56afb33ad1d478b0b4 not found: ID does not exist" containerID="9aa83d6cf7d5df33542ffebf9aac4deb28f9962445b50f56afb33ad1d478b0b4" Oct 30 21:44:18.220489 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:18.220450 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aa83d6cf7d5df33542ffebf9aac4deb28f9962445b50f56afb33ad1d478b0b4"} err="failed to get container status \"9aa83d6cf7d5df33542ffebf9aac4deb28f9962445b50f56afb33ad1d478b0b4\": rpc error: code = NotFound desc = could not find container \"9aa83d6cf7d5df33542ffebf9aac4deb28f9962445b50f56afb33ad1d478b0b4\": container with ID starting with 9aa83d6cf7d5df33542ffebf9aac4deb28f9962445b50f56afb33ad1d478b0b4 not found: ID does not exist" Oct 30 21:44:18.220489 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:18.220466 5801 scope.go:122] "RemoveContainer" containerID="dab09547aefc71ecb03628859f94d70a6cdc6fb33fcbf7092ff40245d7ab568e" Oct 30 21:44:18.234040 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:18.234014 5801 scope.go:122] "RemoveContainer" containerID="dab09547aefc71ecb03628859f94d70a6cdc6fb33fcbf7092ff40245d7ab568e" Oct 30 21:44:18.234359 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:44:18.234324 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dab09547aefc71ecb03628859f94d70a6cdc6fb33fcbf7092ff40245d7ab568e\": container with ID starting with dab09547aefc71ecb03628859f94d70a6cdc6fb33fcbf7092ff40245d7ab568e not found: ID does not exist" containerID="dab09547aefc71ecb03628859f94d70a6cdc6fb33fcbf7092ff40245d7ab568e" Oct 30 21:44:18.234484 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:18.234357 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab09547aefc71ecb03628859f94d70a6cdc6fb33fcbf7092ff40245d7ab568e"} err="failed to get container status \"dab09547aefc71ecb03628859f94d70a6cdc6fb33fcbf7092ff40245d7ab568e\": rpc error: code = NotFound desc = could not find container \"dab09547aefc71ecb03628859f94d70a6cdc6fb33fcbf7092ff40245d7ab568e\": container with ID starting with dab09547aefc71ecb03628859f94d70a6cdc6fb33fcbf7092ff40245d7ab568e not found: ID does not exist" Oct 30 21:44:18.237512 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:18.237487 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6"] Oct 30 21:44:18.242397 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:18.242375 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-484f7514-7c6fcb8dc5-zjgv6"] Oct 30 21:44:18.251656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:18.251631 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp"] Oct 30 21:44:18.256771 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:18.256735 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-484f7514-5b7f7fdf7-876mp"] Oct 30 21:44:18.693031 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:18.692999 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:19.163376 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:19.163344 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:19.265692 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:19.265656 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:19.525234 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:19.525163 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42c85995-b7c3-4a75-9c23-867f10fccb32" path="/var/lib/kubelet/pods/42c85995-b7c3-4a75-9c23-867f10fccb32/volumes" Oct 30 21:44:19.525590 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:19.525575 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a36cd401-1517-4440-b9a5-2a8bab48d14a" path="/var/lib/kubelet/pods/a36cd401-1517-4440-b9a5-2a8bab48d14a/volumes" Oct 30 21:44:19.575220 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:19.575183 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:20.062911 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:20.062874 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:20.348847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:20.348729 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:20.474500 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:20.474460 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:21.009710 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:21.009675 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:21.959998 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:21.959964 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:22.705972 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:22.705930 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:23.076707 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:23.076672 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:23.329568 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:23.329483 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:23.859705 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:23.859668 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:24.150300 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:24.150225 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt" Oct 30 21:44:25.068572 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:25.068533 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:25.085350 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:25.084863 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-685bdabe-hljnv"] Oct 30 21:44:25.093009 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:25.092968 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d80c67-945j6"] Oct 30 21:44:25.098695 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:25.098669 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67e59534-wh8w6"] Oct 30 21:44:25.103885 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:25.103864 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-685bdabe-hljnv"] Oct 30 21:44:25.108846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:25.108820 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d80c67-945j6"] Oct 30 21:44:25.113405 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:25.113383 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67e59534-wh8w6"] Oct 30 21:44:25.272806 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:25.270973 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:25.540864 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:25.540747 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="596fe353-a183-4cb7-9fe7-b4d9cecf90aa" path="/var/lib/kubelet/pods/596fe353-a183-4cb7-9fe7-b4d9cecf90aa/volumes" Oct 30 21:44:25.541317 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:25.541303 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1" path="/var/lib/kubelet/pods/89fb6e5b-34c8-4ef3-a938-fd5f2393eaf1/volumes" Oct 30 21:44:25.541711 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:25.541698 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e18b72ce-2635-4754-9cfb-880e5970a807" path="/var/lib/kubelet/pods/e18b72ce-2635-4754-9cfb-880e5970a807/volumes" Oct 30 21:44:25.597367 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:25.597332 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:26.040050 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.040013 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfad835c-c46dw"] Oct 30 21:44:26.045301 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.045275 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfad835c-c46dw"] Oct 30 21:44:26.066637 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.066604 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:26.300001 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.299908 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:26.419952 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.419918 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:26.491926 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.491891 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft"] Oct 30 21:44:26.493179 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.493150 5801 cpu_manager.go:414] "containerMap: removing container" podUID="42c85995-b7c3-4a75-9c23-867f10fccb32" containerName="payload-pre-processing" Oct 30 21:44:26.493179 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.493178 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42c85995-b7c3-4a75-9c23-867f10fccb32" containerName="payload-pre-processing" Oct 30 21:44:26.493351 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.493211 5801 cpu_manager.go:414] "containerMap: removing container" podUID="a36cd401-1517-4440-b9a5-2a8bab48d14a" containerName="payload-processing" Oct 30 21:44:26.493351 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.493219 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a36cd401-1517-4440-b9a5-2a8bab48d14a" containerName="payload-processing" Oct 30 21:44:26.493445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.493389 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="42c85995-b7c3-4a75-9c23-867f10fccb32" containerName="payload-pre-processing" Oct 30 21:44:26.493445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.493411 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="a36cd401-1517-4440-b9a5-2a8bab48d14a" containerName="payload-processing" Oct 30 21:44:26.508631 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.506525 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.515849 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.515826 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-dc3edf-openshift-default-dockercfg-klcvd\"" Oct 30 21:44:26.520920 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.520892 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft"] Oct 30 21:44:26.565964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.565920 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:26.701773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.694844 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fe17509-a317-4eae-8edd-de1383398615-istio-podinfo\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.701773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.694931 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-workload-socket\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.701773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.695024 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-workload-certs\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.701773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.695066 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-istio-envoy\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.701773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.695114 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fe17509-a317-4eae-8edd-de1383398615-istiod-ca-cert\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.701773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.695159 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-istio-data\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.701773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.695270 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fe17509-a317-4eae-8edd-de1383398615-istio-token\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.701773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.695311 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t9n4\" (UniqueName: \"kubernetes.io/projected/2fe17509-a317-4eae-8edd-de1383398615-kube-api-access-4t9n4\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.701773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.695387 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-credential-socket\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.760232 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.760199 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:26.802884 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.796807 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fe17509-a317-4eae-8edd-de1383398615-istio-podinfo\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.802884 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.796878 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-workload-socket\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.802884 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.796963 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-workload-certs\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.802884 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.797006 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-istio-envoy\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.802884 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.797051 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fe17509-a317-4eae-8edd-de1383398615-istiod-ca-cert\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.802884 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.797090 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-istio-data\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.802884 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.797214 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fe17509-a317-4eae-8edd-de1383398615-istio-token\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.802884 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.797268 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t9n4\" (UniqueName: \"kubernetes.io/projected/2fe17509-a317-4eae-8edd-de1383398615-kube-api-access-4t9n4\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.802884 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.797716 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-workload-socket\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.802884 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.797812 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-credential-socket\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.802884 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.798204 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-credential-socket\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.802884 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.798448 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-workload-certs\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.802884 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.798682 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-istio-data\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.802884 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.801827 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-istio-envoy\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.802884 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.802169 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fe17509-a317-4eae-8edd-de1383398615-istiod-ca-cert\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.809835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.805229 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fe17509-a317-4eae-8edd-de1383398615-istio-podinfo\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.821846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.821137 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t9n4\" (UniqueName: \"kubernetes.io/projected/2fe17509-a317-4eae-8edd-de1383398615-kube-api-access-4t9n4\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.821846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.821207 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fe17509-a317-4eae-8edd-de1383398615-istio-token\") pod \"e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:26.829828 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:26.829215 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:27.147050 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:27.146954 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft"] Oct 30 21:44:27.159000 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:44:27.158938 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe17509_a317_4eae_8edd_de1383398615.slice/crio-876021985e124669c917eb1267ada0d4f073a89a960a96be483f243ddb2c2be8 WatchSource:0}: Error finding container 876021985e124669c917eb1267ada0d4f073a89a960a96be483f243ddb2c2be8: Status 404 returned error can't find the container with id 876021985e124669c917eb1267ada0d4f073a89a960a96be483f243ddb2c2be8 Oct 30 21:44:27.161272 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:27.161225 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:44:27.161394 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:27.161324 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:44:27.161394 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:27.161367 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:44:27.445118 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:27.445031 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" event={"ID":"2fe17509-a317-4eae-8edd-de1383398615","Type":"ContainerStarted","Data":"990d404bc28815dc548b2efe8ed1b456d8f7196f81375af3e75dd80ac7e15524"} Oct 30 21:44:27.445118 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:27.445090 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" event={"ID":"2fe17509-a317-4eae-8edd-de1383398615","Type":"ContainerStarted","Data":"876021985e124669c917eb1267ada0d4f073a89a960a96be483f243ddb2c2be8"} Oct 30 21:44:27.500845 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:27.500789 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:27.518862 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:27.518801 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" podStartSLOduration=1.518780909 podStartE2EDuration="1.518780909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:27.48011853 +0000 UTC m=+936.075935194" watchObservedRunningTime="2026-10-30 21:44:27.518780909 +0000 UTC m=+936.114597571" Oct 30 21:44:27.533845 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:27.533808 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c38d4c28-d335-44ba-991d-37cb69011ca2" path="/var/lib/kubelet/pods/c38d4c28-d335-44ba-991d-37cb69011ca2/volumes" Oct 30 21:44:27.830501 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:27.830455 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:28.270701 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:28.270616 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:28.835378 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:28.835350 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:29.262017 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:29.261933 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:29.476638 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:29.476605 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:29.477487 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:29.477462 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:44:30.082773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:30.077916 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afc89e9a-t8vnz"] Oct 30 21:44:30.094467 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:30.094249 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afc89e9a-t8vnz" Oct 30 21:44:30.106037 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:30.104406 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afc89e9a-t8vnz"] Oct 30 21:44:30.106657 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:30.106631 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:44:30.106819 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:30.106729 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:44:30.107393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:30.107364 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:44:30.107736 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:30.107710 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:44:30.143784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:30.143721 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:30.178254 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:30.178224 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/379f1ea5-497a-49c5-9356-3f402dc3d335-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afc89e9a-t8vnz\" (UID: \"379f1ea5-497a-49c5-9356-3f402dc3d335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afc89e9a-t8vnz" Oct 30 21:44:30.178492 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:30.178336 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl85v\" (UniqueName: \"kubernetes.io/projected/379f1ea5-497a-49c5-9356-3f402dc3d335-kube-api-access-pl85v\") pod \"maas-api-revoke-sub-afc89e9a-t8vnz\" (UID: \"379f1ea5-497a-49c5-9356-3f402dc3d335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afc89e9a-t8vnz" Oct 30 21:44:30.178712 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:30.178688 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/379f1ea5-497a-49c5-9356-3f402dc3d335-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afc89e9a-t8vnz\" (UID: \"379f1ea5-497a-49c5-9356-3f402dc3d335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afc89e9a-t8vnz" Oct 30 21:44:30.279557 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:44:30.279520 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:44:30.280328 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:30.280305 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/379f1ea5-497a-49c5-9356-3f402dc3d335-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afc89e9a-t8vnz\" (UID: \"379f1ea5-497a-49c5-9356-3f402dc3d335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afc89e9a-t8vnz" Oct 30 21:44:30.280463 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:30.280451 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl85v\" (UniqueName: \"kubernetes.io/projected/379f1ea5-497a-49c5-9356-3f402dc3d335-kube-api-access-pl85v\") pod \"maas-api-revoke-sub-afc89e9a-t8vnz\" (UID: \"379f1ea5-497a-49c5-9356-3f402dc3d335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afc89e9a-t8vnz" Oct 30 21:44:30.280662 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:30.280648 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/379f1ea5-497a-49c5-9356-3f402dc3d335-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afc89e9a-t8vnz\" (UID: \"379f1ea5-497a-49c5-9356-3f402dc3d335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afc89e9a-t8vnz" Oct 30 21:44:30.281255 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:30.281234 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/379f1ea5-497a-49c5-9356-3f402dc3d335-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afc89e9a-t8vnz\" (UID: \"379f1ea5-497a-49c5-9356-3f402dc3d335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afc89e9a-t8vnz" Oct 30 21:44:30.282575 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:30.282547 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/379f1ea5-497a-49c5-9356-3f402dc3d335-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afc89e9a-t8vnz\" (UID: \"379f1ea5-497a-49c5-9356-3f402dc3d335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afc89e9a-t8vnz" Oct 30 21:44:30.297951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:30.297931 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl85v\" (UniqueName: \"kubernetes.io/projected/379f1ea5-497a-49c5-9356-3f402dc3d335-kube-api-access-pl85v\") pod \"maas-api-revoke-sub-afc89e9a-t8vnz\" (UID: \"379f1ea5-497a-49c5-9356-3f402dc3d335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afc89e9a-t8vnz" Oct 30 21:44:30.307448 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:30.307419 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:30.431238 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:30.429053 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afc89e9a-t8vnz" Oct 30 21:44:30.447773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:30.435412 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:30.869016 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:44:30.868990 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379f1ea5_497a_49c5_9356_3f402dc3d335.slice/crio-6e59be3dcd5b2fa1a97fe89380158fc5fbbd2dde62b489c093f93664addbe05d WatchSource:0}: Error finding container 6e59be3dcd5b2fa1a97fe89380158fc5fbbd2dde62b489c093f93664addbe05d: Status 404 returned error can't find the container with id 6e59be3dcd5b2fa1a97fe89380158fc5fbbd2dde62b489c093f93664addbe05d Oct 30 21:44:30.870412 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:30.870391 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afc89e9a-t8vnz"] Oct 30 21:44:31.483155 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:31.482857 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:31.581729 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:31.581689 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afc89e9a-t8vnz" event={"ID":"379f1ea5-497a-49c5-9356-3f402dc3d335","Type":"ContainerStarted","Data":"3729e0fa8b1c0b35e967e0f299fbd02d628c80651e073448b3007d5b2108d8a0"} Oct 30 21:44:31.582023 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:31.581985 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afc89e9a-t8vnz" event={"ID":"379f1ea5-497a-49c5-9356-3f402dc3d335","Type":"ContainerStarted","Data":"6e59be3dcd5b2fa1a97fe89380158fc5fbbd2dde62b489c093f93664addbe05d"} Oct 30 21:44:31.593813 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:31.593787 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:31.607463 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:31.607413 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afc89e9a-t8vnz" podStartSLOduration=1.607398565 podStartE2EDuration="1.607398565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:31.6034778 +0000 UTC m=+940.199294461" watchObservedRunningTime="2026-10-30 21:44:31.607398565 +0000 UTC m=+940.203215227" Oct 30 21:44:32.393508 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:32.393466 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:32.593407 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:32.593370 5801 generic.go:356] "Generic (PLEG): container finished" podID="379f1ea5-497a-49c5-9356-3f402dc3d335" containerID="3729e0fa8b1c0b35e967e0f299fbd02d628c80651e073448b3007d5b2108d8a0" exitCode=0 Oct 30 21:44:32.593821 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:32.593451 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afc89e9a-t8vnz" event={"ID":"379f1ea5-497a-49c5-9356-3f402dc3d335","Type":"ContainerDied","Data":"3729e0fa8b1c0b35e967e0f299fbd02d628c80651e073448b3007d5b2108d8a0"} Oct 30 21:44:33.427776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:33.423025 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:33.805469 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:33.805445 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afc89e9a-t8vnz" Oct 30 21:44:33.883670 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:33.883633 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/379f1ea5-497a-49c5-9356-3f402dc3d335-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/379f1ea5-497a-49c5-9356-3f402dc3d335-maas-api-openshift-ca\") pod \"379f1ea5-497a-49c5-9356-3f402dc3d335\" (UID: \"379f1ea5-497a-49c5-9356-3f402dc3d335\") " Oct 30 21:44:33.883882 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:33.883685 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/379f1ea5-497a-49c5-9356-3f402dc3d335-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/379f1ea5-497a-49c5-9356-3f402dc3d335-maas-api-xks-ca\") pod \"379f1ea5-497a-49c5-9356-3f402dc3d335\" (UID: \"379f1ea5-497a-49c5-9356-3f402dc3d335\") " Oct 30 21:44:33.883882 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:33.883798 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/379f1ea5-497a-49c5-9356-3f402dc3d335-kube-api-access-pl85v\" (UniqueName: \"kubernetes.io/projected/379f1ea5-497a-49c5-9356-3f402dc3d335-kube-api-access-pl85v\") pod \"379f1ea5-497a-49c5-9356-3f402dc3d335\" (UID: \"379f1ea5-497a-49c5-9356-3f402dc3d335\") " Oct 30 21:44:33.884086 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:33.884061 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/379f1ea5-497a-49c5-9356-3f402dc3d335-maas-api-openshift-ca" pod "379f1ea5-497a-49c5-9356-3f402dc3d335" (UID: "379f1ea5-497a-49c5-9356-3f402dc3d335"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:33.884464 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:33.884427 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/379f1ea5-497a-49c5-9356-3f402dc3d335-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:33.885966 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:33.885938 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/379f1ea5-497a-49c5-9356-3f402dc3d335-kube-api-access-pl85v" pod "379f1ea5-497a-49c5-9356-3f402dc3d335" (UID: "379f1ea5-497a-49c5-9356-3f402dc3d335"). InnerVolumeSpecName "kube-api-access-pl85v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:33.886369 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:33.886347 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/379f1ea5-497a-49c5-9356-3f402dc3d335-maas-api-xks-ca" pod "379f1ea5-497a-49c5-9356-3f402dc3d335" (UID: "379f1ea5-497a-49c5-9356-3f402dc3d335"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:33.985939 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:33.985901 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/379f1ea5-497a-49c5-9356-3f402dc3d335-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:33.985939 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:33.985935 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl85v\" (UniqueName: \"kubernetes.io/projected/379f1ea5-497a-49c5-9356-3f402dc3d335-kube-api-access-pl85v\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:34.063416 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:34.062610 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:34.624818 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:34.624784 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afc89e9a-t8vnz" event={"ID":"379f1ea5-497a-49c5-9356-3f402dc3d335","Type":"ContainerDied","Data":"6e59be3dcd5b2fa1a97fe89380158fc5fbbd2dde62b489c093f93664addbe05d"} Oct 30 21:44:34.624996 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:34.624839 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e59be3dcd5b2fa1a97fe89380158fc5fbbd2dde62b489c093f93664addbe05d" Oct 30 21:44:34.624996 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:34.624791 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afc89e9a-t8vnz" Oct 30 21:44:34.652980 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:34.652948 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:35.031540 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:35.031430 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4665d99f-qbb5m"] Oct 30 21:44:35.038106 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:35.038080 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4665d99f-qbb5m"] Oct 30 21:44:35.253266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:35.253232 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:35.528796 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:35.528744 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6512e893-ad04-45d9-bdd9-63bf8a5aec2f" path="/var/lib/kubelet/pods/6512e893-ad04-45d9-bdd9-63bf8a5aec2f/volumes" Oct 30 21:44:35.577222 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:35.577187 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:35.691901 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:35.691865 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:35.809483 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:35.809437 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:35.906815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:35.906781 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:36.160939 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.160850 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq"] Oct 30 21:44:36.161542 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.161510 5801 cpu_manager.go:414] "containerMap: removing container" podUID="379f1ea5-497a-49c5-9356-3f402dc3d335" containerName="revoke-keys" Oct 30 21:44:36.161542 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.161527 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="379f1ea5-497a-49c5-9356-3f402dc3d335" containerName="revoke-keys" Oct 30 21:44:36.161703 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.161689 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="379f1ea5-497a-49c5-9356-3f402dc3d335" containerName="revoke-keys" Oct 30 21:44:36.166981 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.166957 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.170862 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.170843 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-fbf428-openshift-default-dockercfg-cqlbx\"" Oct 30 21:44:36.201037 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.201005 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq"] Oct 30 21:44:36.222030 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.221989 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:36.236106 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.236070 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-data\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.236254 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.236116 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-podinfo\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.236321 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.236296 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-token\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.236381 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.236361 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-workload-socket\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.236499 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.236482 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-credential-socket\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.236603 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.236588 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgpht\" (UniqueName: \"kubernetes.io/projected/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-kube-api-access-qgpht\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.236668 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.236650 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istiod-ca-cert\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.236823 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.236805 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-workload-certs\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.236869 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.236851 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-envoy\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.337960 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.337926 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istiod-ca-cert\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.338131 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.338011 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-workload-certs\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.338131 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.338050 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-envoy\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.338131 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.338121 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-data\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.338295 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.338139 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-podinfo\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.338295 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.338196 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-token\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.338295 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.338231 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-workload-socket\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.338447 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.338307 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-credential-socket\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.338447 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.338391 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgpht\" (UniqueName: \"kubernetes.io/projected/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-kube-api-access-qgpht\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.338564 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.338435 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-workload-certs\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.338564 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.338553 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-data\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.338714 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.338692 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istiod-ca-cert\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.338843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.338746 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-credential-socket\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.338917 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.338859 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-workload-socket\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.340436 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.340411 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-envoy\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.340649 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.340630 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-podinfo\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.350257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.350237 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-token\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.350592 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.350572 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgpht\" (UniqueName: \"kubernetes.io/projected/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-kube-api-access-qgpht\") pod \"e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.355931 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.355902 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:36.484047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.482736 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:36.765968 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.765806 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:36.927436 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:44:36.927403 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea737f7_f6dc_4b40_8c4c_920a2131fcbe.slice/crio-89337231b67d5787e60862ab482ed3cbe94586940128510255e53fb15b8c86c5 WatchSource:0}: Error finding container 89337231b67d5787e60862ab482ed3cbe94586940128510255e53fb15b8c86c5: Status 404 returned error can't find the container with id 89337231b67d5787e60862ab482ed3cbe94586940128510255e53fb15b8c86c5 Oct 30 21:44:36.929494 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.929471 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq"] Oct 30 21:44:36.930474 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.930446 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:44:36.930548 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.930512 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:44:36.930548 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:36.930542 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:44:37.658463 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:37.658420 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" event={"ID":"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe","Type":"ContainerStarted","Data":"afdce3adb37cb696660601a7973845154500045189525ba635e7bc274a0c5636"} Oct 30 21:44:37.658894 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:37.658488 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" event={"ID":"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe","Type":"ContainerStarted","Data":"89337231b67d5787e60862ab482ed3cbe94586940128510255e53fb15b8c86c5"} Oct 30 21:44:37.676418 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:37.676381 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:37.705118 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:37.705063 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" podStartSLOduration=1.705047454 podStartE2EDuration="1.705047454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:37.701942529 +0000 UTC m=+946.297759191" watchObservedRunningTime="2026-10-30 21:44:37.705047454 +0000 UTC m=+946.300864113" Oct 30 21:44:38.066860 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:38.066821 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a86bb3c6-r4pzd"] Oct 30 21:44:38.073271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:38.073230 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a86bb3c6-r4pzd"] Oct 30 21:44:38.444261 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:38.444173 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:38.486457 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:38.485169 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:38.500825 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:38.498391 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:38.668827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:38.668792 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:38.669742 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:38.669722 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:44:39.472650 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:39.472612 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:39.532746 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:39.532711 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b503661-236a-483a-9395-53a63284643e" path="/var/lib/kubelet/pods/2b503661-236a-483a-9395-53a63284643e/volumes" Oct 30 21:44:40.144843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:40.144800 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:40.335002 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:40.334969 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:40.622698 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:40.622667 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:40.729621 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:40.729590 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:41.411478 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.411441 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:41.470557 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.470525 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw"] Oct 30 21:44:41.475620 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.475594 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw" Oct 30 21:44:41.492879 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.491312 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-fbf428\"" Oct 30 21:44:41.505843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.504861 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw"] Oct 30 21:44:41.542858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.542814 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx"] Oct 30 21:44:41.556519 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.556475 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:41.558508 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.558471 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx" Oct 30 21:44:41.565589 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.565563 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-fbf428-dockercfg-tx5rp\"" Oct 30 21:44:41.565830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.565806 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9fqf\" (UniqueName: \"kubernetes.io/projected/a5559491-62e9-4637-9253-ac2edfccbd46-kube-api-access-b9fqf\") pod \"payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw\" (UID: \"a5559491-62e9-4637-9253-ac2edfccbd46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw" Oct 30 21:44:41.565992 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.565975 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5559491-62e9-4637-9253-ac2edfccbd46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw\" (UID: \"a5559491-62e9-4637-9253-ac2edfccbd46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw" Oct 30 21:44:41.572405 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.572282 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx"] Oct 30 21:44:41.698829 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.676473 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5559491-62e9-4637-9253-ac2edfccbd46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw\" (UID: \"a5559491-62e9-4637-9253-ac2edfccbd46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw" Oct 30 21:44:41.698829 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.676618 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb686228-0c5b-4d64-82b4-1838e0f75369-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx\" (UID: \"cb686228-0c5b-4d64-82b4-1838e0f75369\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx" Oct 30 21:44:41.698829 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.676958 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p97xk\" (UniqueName: \"kubernetes.io/projected/cb686228-0c5b-4d64-82b4-1838e0f75369-kube-api-access-p97xk\") pod \"payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx\" (UID: \"cb686228-0c5b-4d64-82b4-1838e0f75369\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx" Oct 30 21:44:41.698829 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.677006 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9fqf\" (UniqueName: \"kubernetes.io/projected/a5559491-62e9-4637-9253-ac2edfccbd46-kube-api-access-b9fqf\") pod \"payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw\" (UID: \"a5559491-62e9-4637-9253-ac2edfccbd46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw" Oct 30 21:44:41.698829 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.679657 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5559491-62e9-4637-9253-ac2edfccbd46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw\" (UID: \"a5559491-62e9-4637-9253-ac2edfccbd46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw" Oct 30 21:44:41.731652 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.729916 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9fqf\" (UniqueName: \"kubernetes.io/projected/a5559491-62e9-4637-9253-ac2edfccbd46-kube-api-access-b9fqf\") pod \"payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw\" (UID: \"a5559491-62e9-4637-9253-ac2edfccbd46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw" Oct 30 21:44:41.748667 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.748621 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:41.779182 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.779152 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb686228-0c5b-4d64-82b4-1838e0f75369-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx\" (UID: \"cb686228-0c5b-4d64-82b4-1838e0f75369\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx" Oct 30 21:44:41.779555 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.779537 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p97xk\" (UniqueName: \"kubernetes.io/projected/cb686228-0c5b-4d64-82b4-1838e0f75369-kube-api-access-p97xk\") pod \"payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx\" (UID: \"cb686228-0c5b-4d64-82b4-1838e0f75369\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx" Oct 30 21:44:41.780377 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.780350 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb686228-0c5b-4d64-82b4-1838e0f75369-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx\" (UID: \"cb686228-0c5b-4d64-82b4-1838e0f75369\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx" Oct 30 21:44:41.798431 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.798402 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p97xk\" (UniqueName: \"kubernetes.io/projected/cb686228-0c5b-4d64-82b4-1838e0f75369-kube-api-access-p97xk\") pod \"payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx\" (UID: \"cb686228-0c5b-4d64-82b4-1838e0f75369\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx" Oct 30 21:44:41.816357 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.816326 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw" Oct 30 21:44:41.910209 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:41.899262 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx" Oct 30 21:44:42.160705 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:42.160670 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw"] Oct 30 21:44:42.336333 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:44:42.336308 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb686228_0c5b_4d64_82b4_1838e0f75369.slice/crio-1b07fbdab8e7cec271ab5e7faa0d469f48410299527edb1b1ac15110174b6a94 WatchSource:0}: Error finding container 1b07fbdab8e7cec271ab5e7faa0d469f48410299527edb1b1ac15110174b6a94: Status 404 returned error can't find the container with id 1b07fbdab8e7cec271ab5e7faa0d469f48410299527edb1b1ac15110174b6a94 Oct 30 21:44:42.338091 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:42.338071 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx"] Oct 30 21:44:42.657783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:42.657685 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:42.796153 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:42.796117 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx" event={"ID":"cb686228-0c5b-4d64-82b4-1838e0f75369","Type":"ContainerStarted","Data":"e9f60dbe44c7866b366e1f6c1904473a4a30b1d870af01a0b4859a15071fb986"} Oct 30 21:44:42.796371 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:42.796182 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx" event={"ID":"cb686228-0c5b-4d64-82b4-1838e0f75369","Type":"ContainerStarted","Data":"1b07fbdab8e7cec271ab5e7faa0d469f48410299527edb1b1ac15110174b6a94"} Oct 30 21:44:42.796551 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:42.796521 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx" Oct 30 21:44:42.805011 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:42.804971 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw" event={"ID":"a5559491-62e9-4637-9253-ac2edfccbd46","Type":"ContainerStarted","Data":"2d7209188df30f2692627a27618727640e7899b327132ac72e6221645bc6c72a"} Oct 30 21:44:42.805201 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:42.805036 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw" event={"ID":"a5559491-62e9-4637-9253-ac2edfccbd46","Type":"ContainerStarted","Data":"ce9b64a19201d1fc6aa341617ab1367af68faa814c0b98ccec8a167002156235"} Oct 30 21:44:42.805257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:42.805237 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw" Oct 30 21:44:42.824623 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:42.824575 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx" podStartSLOduration=1.824560559 podStartE2EDuration="1.824560559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:42.820902479 +0000 UTC m=+951.416719141" watchObservedRunningTime="2026-10-30 21:44:42.824560559 +0000 UTC m=+951.420377218" Oct 30 21:44:42.855909 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:42.855860 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw" podStartSLOduration=1.855847535 podStartE2EDuration="1.855847535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:42.850744604 +0000 UTC m=+951.446561275" watchObservedRunningTime="2026-10-30 21:44:42.855847535 +0000 UTC m=+951.451664195" Oct 30 21:44:43.475478 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:43.475443 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:44.294393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:44.294361 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:45.070830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:45.070792 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:45.959501 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:45.959462 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:46.062292 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:46.062259 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:46.786339 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:46.786288 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:47.783654 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:47.766888 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:47.938560 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:47.938525 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:48.728889 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:48.728855 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:49.676125 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:49.676075 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:50.408254 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:50.408180 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:50.887954 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:50.887922 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:50.992060 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:50.992020 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:51.403715 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.403679 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69"] Oct 30 21:44:51.409639 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.409615 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.419193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.419168 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-0c62ab-openshift-default-dockercfg-qhbsg\"" Oct 30 21:44:51.428697 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.428638 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69"] Oct 30 21:44:51.439203 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.439171 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:51.476612 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.476564 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-credential-socket\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.476809 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.476646 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-token\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.476870 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.476830 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/457b1dc9-f94b-4dda-8b46-e125e0069033-istiod-ca-cert\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.476985 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.476965 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-podinfo\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.477120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.477099 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-data\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.477175 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.477159 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc8sh\" (UniqueName: \"kubernetes.io/projected/457b1dc9-f94b-4dda-8b46-e125e0069033-kube-api-access-rc8sh\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.477327 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.477307 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-envoy\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.477447 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.477372 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-workload-socket\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.477500 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.477479 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-workload-certs\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.548530 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.548495 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:51.579318 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.579281 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/457b1dc9-f94b-4dda-8b46-e125e0069033-istiod-ca-cert\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.579517 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.579369 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-podinfo\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.579517 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.579492 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-data\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.579652 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.579549 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc8sh\" (UniqueName: \"kubernetes.io/projected/457b1dc9-f94b-4dda-8b46-e125e0069033-kube-api-access-rc8sh\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.579717 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.579674 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-envoy\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.579820 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.579740 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-workload-socket\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.579892 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.579838 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-workload-certs\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.579980 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.579945 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-credential-socket\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.580043 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.580010 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-token\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.580171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.579945 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-data\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.580171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.580142 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-workload-socket\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.580171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.580165 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-workload-certs\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.580348 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.580205 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/457b1dc9-f94b-4dda-8b46-e125e0069033-istiod-ca-cert\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.580348 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.580230 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-credential-socket\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.581972 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.581948 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-envoy\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.582071 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.582040 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-podinfo\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.592014 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.591991 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc8sh\" (UniqueName: \"kubernetes.io/projected/457b1dc9-f94b-4dda-8b46-e125e0069033-kube-api-access-rc8sh\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.595346 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.595314 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-token\") pod \"e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.755830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.755061 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-0c62ab-openshift-default-dockercfg-qhbsg\"" Oct 30 21:44:51.766321 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.764925 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:51.951802 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:51.950656 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:52.139467 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:52.139418 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:44:52.141939 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:52.141907 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:44:52.142175 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:52.142154 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:44:52.149844 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:52.149410 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69"] Oct 30 21:44:52.887100 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:52.887061 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:52.946699 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:52.946664 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" event={"ID":"457b1dc9-f94b-4dda-8b46-e125e0069033","Type":"ContainerStarted","Data":"0046d3214c1887638295980b75d32b3ee903884b4ca8bbcb13b97e894e8aca52"} Oct 30 21:44:52.946900 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:52.946728 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" event={"ID":"457b1dc9-f94b-4dda-8b46-e125e0069033","Type":"ContainerStarted","Data":"10d7bbd534188a89c44218e72a866bc0726c385679b92e04d99a2668856deb12"} Oct 30 21:44:52.976142 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:52.976069 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" podStartSLOduration=1.976050217 podStartE2EDuration="1.976050217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:52.973993521 +0000 UTC m=+961.569810182" watchObservedRunningTime="2026-10-30 21:44:52.976050217 +0000 UTC m=+961.571866889" Oct 30 21:44:53.768775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:53.766675 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:53.804093 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:53.804061 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:53.846578 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:53.846328 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx" Oct 30 21:44:53.846818 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:53.846638 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw" Oct 30 21:44:53.927292 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:53.927257 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:53.962575 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:53.962538 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:53.963637 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:53.963609 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:44:54.950690 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:54.950646 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:55.395167 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:55.395127 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:55.532352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:55.532316 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:56.045944 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:56.045909 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:56.366319 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:56.366288 5801 scope.go:122] "RemoveContainer" containerID="90378d517d91838283de2b675dc87a9d89efd80603734700fae672061ee1a964" Oct 30 21:44:56.378349 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:56.378323 5801 scope.go:122] "RemoveContainer" containerID="62597f6ec99dd8bface1e458a080a36620ceb87dfce0645ddf90987701cef2c1" Oct 30 21:44:56.389160 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:56.389139 5801 scope.go:122] "RemoveContainer" containerID="a9a0706021e78655a789b05137af810a13ae9f7da6cb867a78cc20df28f4b96f" Oct 30 21:44:56.878276 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:56.878238 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:56.937481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:56.937444 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5"] Oct 30 21:44:56.944321 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:56.944292 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5" Oct 30 21:44:56.954074 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:56.954021 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-0c62ab\"" Oct 30 21:44:56.954840 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:56.954816 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-0c62ab-dockercfg-gsk4v\"" Oct 30 21:44:56.958425 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:56.958397 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5"] Oct 30 21:44:56.995165 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:56.995131 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k"] Oct 30 21:44:56.995340 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:56.995274 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb967\" (UniqueName: \"kubernetes.io/projected/00c8f036-7f24-4c36-94c4-a09a666338a2-kube-api-access-kb967\") pod \"payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5\" (UID: \"00c8f036-7f24-4c36-94c4-a09a666338a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5" Oct 30 21:44:56.995404 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:56.995336 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c8f036-7f24-4c36-94c4-a09a666338a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5\" (UID: \"00c8f036-7f24-4c36-94c4-a09a666338a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5" Oct 30 21:44:57.000642 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:57.000612 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k" Oct 30 21:44:57.013421 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:57.013385 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k"] Oct 30 21:44:57.015336 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:57.015125 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:57.137826 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:57.131003 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g5qv\" (UniqueName: \"kubernetes.io/projected/35404e22-3c01-4e3c-bcf6-960df3dadfe0-kube-api-access-2g5qv\") pod \"payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k\" (UID: \"35404e22-3c01-4e3c-bcf6-960df3dadfe0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k" Oct 30 21:44:57.137826 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:57.131088 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb967\" (UniqueName: \"kubernetes.io/projected/00c8f036-7f24-4c36-94c4-a09a666338a2-kube-api-access-kb967\") pod \"payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5\" (UID: \"00c8f036-7f24-4c36-94c4-a09a666338a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5" Oct 30 21:44:57.137826 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:57.131144 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c8f036-7f24-4c36-94c4-a09a666338a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5\" (UID: \"00c8f036-7f24-4c36-94c4-a09a666338a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5" Oct 30 21:44:57.137826 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:57.131194 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35404e22-3c01-4e3c-bcf6-960df3dadfe0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k\" (UID: \"35404e22-3c01-4e3c-bcf6-960df3dadfe0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k" Oct 30 21:44:57.190786 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:57.189481 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:57.205834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:57.204818 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c8f036-7f24-4c36-94c4-a09a666338a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5\" (UID: \"00c8f036-7f24-4c36-94c4-a09a666338a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5" Oct 30 21:44:57.285812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:57.247211 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g5qv\" (UniqueName: \"kubernetes.io/projected/35404e22-3c01-4e3c-bcf6-960df3dadfe0-kube-api-access-2g5qv\") pod \"payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k\" (UID: \"35404e22-3c01-4e3c-bcf6-960df3dadfe0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k" Oct 30 21:44:57.285812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:57.247449 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35404e22-3c01-4e3c-bcf6-960df3dadfe0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k\" (UID: \"35404e22-3c01-4e3c-bcf6-960df3dadfe0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k" Oct 30 21:44:57.285812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:57.247999 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb967\" (UniqueName: \"kubernetes.io/projected/00c8f036-7f24-4c36-94c4-a09a666338a2-kube-api-access-kb967\") pod \"payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5\" (UID: \"00c8f036-7f24-4c36-94c4-a09a666338a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5" Oct 30 21:44:57.285812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:57.248781 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35404e22-3c01-4e3c-bcf6-960df3dadfe0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k\" (UID: \"35404e22-3c01-4e3c-bcf6-960df3dadfe0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k" Oct 30 21:44:57.285812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:57.278210 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5" Oct 30 21:44:57.345325 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:57.343839 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g5qv\" (UniqueName: \"kubernetes.io/projected/35404e22-3c01-4e3c-bcf6-960df3dadfe0-kube-api-access-2g5qv\") pod \"payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k\" (UID: \"35404e22-3c01-4e3c-bcf6-960df3dadfe0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k" Oct 30 21:44:57.363567 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:57.363538 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k" Oct 30 21:44:57.623855 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:57.623825 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:57.692492 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:44:57.692450 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c8f036_7f24_4c36_94c4_a09a666338a2.slice/crio-9dc573a00ce9df95a49757a2e6f491b41ceb7d7198b051e42dad0fc3a7369a22 WatchSource:0}: Error finding container 9dc573a00ce9df95a49757a2e6f491b41ceb7d7198b051e42dad0fc3a7369a22: Status 404 returned error can't find the container with id 9dc573a00ce9df95a49757a2e6f491b41ceb7d7198b051e42dad0fc3a7369a22 Oct 30 21:44:57.694726 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:57.694704 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5"] Oct 30 21:44:57.725054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:57.725028 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:57.726954 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:57.726928 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k"] Oct 30 21:44:57.730125 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:44:57.730100 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35404e22_3c01_4e3c_bcf6_960df3dadfe0.slice/crio-647499fe09b13260c69863f6c27c647d2945ccda6c8d7be8b95fd63ba54bf2b5 WatchSource:0}: Error finding container 647499fe09b13260c69863f6c27c647d2945ccda6c8d7be8b95fd63ba54bf2b5: Status 404 returned error can't find the container with id 647499fe09b13260c69863f6c27c647d2945ccda6c8d7be8b95fd63ba54bf2b5 Oct 30 21:44:58.015834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:58.015697 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k" event={"ID":"35404e22-3c01-4e3c-bcf6-960df3dadfe0","Type":"ContainerStarted","Data":"0647854103e1e7c22bd59ce00da9b50ca2024503bcfc69f3c5b3e28ff2cda9d2"} Oct 30 21:44:58.015834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:58.015798 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k" event={"ID":"35404e22-3c01-4e3c-bcf6-960df3dadfe0","Type":"ContainerStarted","Data":"647499fe09b13260c69863f6c27c647d2945ccda6c8d7be8b95fd63ba54bf2b5"} Oct 30 21:44:58.016129 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:58.015967 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k" Oct 30 21:44:58.018519 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:58.018463 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5" event={"ID":"00c8f036-7f24-4c36-94c4-a09a666338a2","Type":"ContainerStarted","Data":"9ca757fbec5d607fde6a87feb184c7f4843f463daf594fa70df1d11c55b31328"} Oct 30 21:44:58.018674 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:58.018531 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5" Oct 30 21:44:58.018674 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:58.018554 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5" event={"ID":"00c8f036-7f24-4c36-94c4-a09a666338a2","Type":"ContainerStarted","Data":"9dc573a00ce9df95a49757a2e6f491b41ceb7d7198b051e42dad0fc3a7369a22"} Oct 30 21:44:58.059931 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:58.059883 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k" podStartSLOduration=2.059869945 podStartE2EDuration="2.059869945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:58.053701277 +0000 UTC m=+966.649517940" watchObservedRunningTime="2026-10-30 21:44:58.059869945 +0000 UTC m=+966.655686604" Oct 30 21:44:58.078823 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:58.078788 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:58.088357 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:58.088297 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5" podStartSLOduration=2.088278583 podStartE2EDuration="2.088278583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:58.084798437 +0000 UTC m=+966.680615121" watchObservedRunningTime="2026-10-30 21:44:58.088278583 +0000 UTC m=+966.684095245" Oct 30 21:44:58.196448 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:58.196413 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:58.932768 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:58.932713 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:59.052480 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:59.052439 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:59.872656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:44:59.872616 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:00.176853 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.176766 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9"] Oct 30 21:45:00.196301 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.196272 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9" Oct 30 21:45:00.197517 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.197357 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52"] Oct 30 21:45:00.207197 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.206286 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8"] Oct 30 21:45:00.207197 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.207124 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52" Oct 30 21:45:00.213087 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.211934 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8" Oct 30 21:45:00.222701 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.219937 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb"] Oct 30 21:45:00.279305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.279276 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb" Oct 30 21:45:00.295839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.285719 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:45:00.295839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.286365 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:45:00.295839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.294618 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:45:00.328675 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.310108 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp"] Oct 30 21:45:00.333874 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:00.332335 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:45:00.334475 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.334217 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9"] Oct 30 21:45:00.334475 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.334261 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52"] Oct 30 21:45:00.334475 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.334360 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp" Oct 30 21:45:00.347211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.346443 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8"] Oct 30 21:45:00.357771 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.355598 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8"] Oct 30 21:45:00.369256 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.369217 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4314f13-77fd-48d6-9387-3648850025da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8\" (UID: \"e4314f13-77fd-48d6-9387-3648850025da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8" Oct 30 21:45:00.369387 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.369279 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlwb8\" (UniqueName: \"kubernetes.io/projected/e4314f13-77fd-48d6-9387-3648850025da-kube-api-access-nlwb8\") pod \"maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8\" (UID: \"e4314f13-77fd-48d6-9387-3648850025da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8" Oct 30 21:45:00.369387 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.369380 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f58a71e2-3701-43d1-a341-58e5bef8732d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52\" (UID: \"f58a71e2-3701-43d1-a341-58e5bef8732d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52" Oct 30 21:45:00.369536 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.369517 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51fc9f9d-9a00-4001-b258-7f7850297a0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9\" (UID: \"51fc9f9d-9a00-4001-b258-7f7850297a0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9" Oct 30 21:45:00.369677 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.369660 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmdnx\" (UniqueName: \"kubernetes.io/projected/f58a71e2-3701-43d1-a341-58e5bef8732d-kube-api-access-mmdnx\") pod \"maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52\" (UID: \"f58a71e2-3701-43d1-a341-58e5bef8732d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52" Oct 30 21:45:00.369838 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.369801 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xjp4\" (UniqueName: \"kubernetes.io/projected/51fc9f9d-9a00-4001-b258-7f7850297a0e-kube-api-access-2xjp4\") pod \"maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9\" (UID: \"51fc9f9d-9a00-4001-b258-7f7850297a0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9" Oct 30 21:45:00.377378 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.377347 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8" Oct 30 21:45:00.378864 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.378839 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp"] Oct 30 21:45:00.392887 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.392827 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6"] Oct 30 21:45:00.398397 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.398377 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6" Oct 30 21:45:00.400455 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.400421 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb"] Oct 30 21:45:00.406363 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.406347 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6"] Oct 30 21:45:00.412390 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.412367 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8"] Oct 30 21:45:00.418165 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.418146 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw"] Oct 30 21:45:00.422856 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.422838 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw" Oct 30 21:45:00.423045 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.423024 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286"] Oct 30 21:45:00.428384 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.428334 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286" Oct 30 21:45:00.429713 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.429690 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286"] Oct 30 21:45:00.436029 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.436009 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw"] Oct 30 21:45:00.472845 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.472803 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ad7312-8811-476b-9e22-2d062cfa5508-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp\" (UID: \"b4ad7312-8811-476b-9e22-2d062cfa5508\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp" Oct 30 21:45:00.473044 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.472906 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51fc9f9d-9a00-4001-b258-7f7850297a0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9\" (UID: \"51fc9f9d-9a00-4001-b258-7f7850297a0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9" Oct 30 21:45:00.473044 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.473014 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmdnx\" (UniqueName: \"kubernetes.io/projected/f58a71e2-3701-43d1-a341-58e5bef8732d-kube-api-access-mmdnx\") pod \"maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52\" (UID: \"f58a71e2-3701-43d1-a341-58e5bef8732d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52" Oct 30 21:45:00.473203 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.473083 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5nzz\" (UniqueName: \"kubernetes.io/projected/e76f09d3-4c42-4917-92cb-0f61d7cb381e-kube-api-access-x5nzz\") pod \"maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb\" (UID: \"e76f09d3-4c42-4917-92cb-0f61d7cb381e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb" Oct 30 21:45:00.473203 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.473136 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xjp4\" (UniqueName: \"kubernetes.io/projected/51fc9f9d-9a00-4001-b258-7f7850297a0e-kube-api-access-2xjp4\") pod \"maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9\" (UID: \"51fc9f9d-9a00-4001-b258-7f7850297a0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9" Oct 30 21:45:00.473203 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.473157 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e76f09d3-4c42-4917-92cb-0f61d7cb381e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb\" (UID: \"e76f09d3-4c42-4917-92cb-0f61d7cb381e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb" Oct 30 21:45:00.473203 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.473199 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vkbb\" (UniqueName: \"kubernetes.io/projected/b4ad7312-8811-476b-9e22-2d062cfa5508-kube-api-access-5vkbb\") pod \"maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp\" (UID: \"b4ad7312-8811-476b-9e22-2d062cfa5508\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp" Oct 30 21:45:00.473457 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.473240 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4314f13-77fd-48d6-9387-3648850025da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8\" (UID: \"e4314f13-77fd-48d6-9387-3648850025da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8" Oct 30 21:45:00.473457 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.473262 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlwb8\" (UniqueName: \"kubernetes.io/projected/e4314f13-77fd-48d6-9387-3648850025da-kube-api-access-nlwb8\") pod \"maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8\" (UID: \"e4314f13-77fd-48d6-9387-3648850025da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8" Oct 30 21:45:00.473457 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.473285 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx8sq\" (UniqueName: \"kubernetes.io/projected/863ab859-100b-4fe9-843c-8390ebeab018-kube-api-access-mx8sq\") pod \"maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8\" (UID: \"863ab859-100b-4fe9-843c-8390ebeab018\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8" Oct 30 21:45:00.473457 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.473306 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/863ab859-100b-4fe9-843c-8390ebeab018-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8\" (UID: \"863ab859-100b-4fe9-843c-8390ebeab018\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8" Oct 30 21:45:00.473457 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.473358 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f58a71e2-3701-43d1-a341-58e5bef8732d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52\" (UID: \"f58a71e2-3701-43d1-a341-58e5bef8732d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52" Oct 30 21:45:00.473975 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.473948 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51fc9f9d-9a00-4001-b258-7f7850297a0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9\" (UID: \"51fc9f9d-9a00-4001-b258-7f7850297a0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9" Oct 30 21:45:00.473975 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.473969 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f58a71e2-3701-43d1-a341-58e5bef8732d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52\" (UID: \"f58a71e2-3701-43d1-a341-58e5bef8732d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52" Oct 30 21:45:00.474228 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.474205 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4314f13-77fd-48d6-9387-3648850025da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8\" (UID: \"e4314f13-77fd-48d6-9387-3648850025da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8" Oct 30 21:45:00.490051 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.490022 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xjp4\" (UniqueName: \"kubernetes.io/projected/51fc9f9d-9a00-4001-b258-7f7850297a0e-kube-api-access-2xjp4\") pod \"maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9\" (UID: \"51fc9f9d-9a00-4001-b258-7f7850297a0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9" Oct 30 21:45:00.490202 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.490022 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmdnx\" (UniqueName: \"kubernetes.io/projected/f58a71e2-3701-43d1-a341-58e5bef8732d-kube-api-access-mmdnx\") pod \"maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52\" (UID: \"f58a71e2-3701-43d1-a341-58e5bef8732d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52" Oct 30 21:45:00.490202 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.490149 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlwb8\" (UniqueName: \"kubernetes.io/projected/e4314f13-77fd-48d6-9387-3648850025da-kube-api-access-nlwb8\") pod \"maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8\" (UID: \"e4314f13-77fd-48d6-9387-3648850025da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8" Oct 30 21:45:00.575196 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.575153 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15e03e4b-deae-4513-83d2-efd7f5538d24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw\" (UID: \"15e03e4b-deae-4513-83d2-efd7f5538d24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw" Oct 30 21:45:00.575396 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.575209 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5nzz\" (UniqueName: \"kubernetes.io/projected/e76f09d3-4c42-4917-92cb-0f61d7cb381e-kube-api-access-x5nzz\") pod \"maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb\" (UID: \"e76f09d3-4c42-4917-92cb-0f61d7cb381e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb" Oct 30 21:45:00.575396 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.575358 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e76f09d3-4c42-4917-92cb-0f61d7cb381e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb\" (UID: \"e76f09d3-4c42-4917-92cb-0f61d7cb381e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb" Oct 30 21:45:00.575530 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.575397 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/245e03b0-e13d-42fd-a32f-33891baa7f44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6\" (UID: \"245e03b0-e13d-42fd-a32f-33891baa7f44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6" Oct 30 21:45:00.575530 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.575454 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vkbb\" (UniqueName: \"kubernetes.io/projected/b4ad7312-8811-476b-9e22-2d062cfa5508-kube-api-access-5vkbb\") pod \"maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp\" (UID: \"b4ad7312-8811-476b-9e22-2d062cfa5508\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp" Oct 30 21:45:00.575649 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.575543 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw8sm\" (UniqueName: \"kubernetes.io/projected/245e03b0-e13d-42fd-a32f-33891baa7f44-kube-api-access-kw8sm\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6\" (UID: \"245e03b0-e13d-42fd-a32f-33891baa7f44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6" Oct 30 21:45:00.575649 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.575585 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx8sq\" (UniqueName: \"kubernetes.io/projected/863ab859-100b-4fe9-843c-8390ebeab018-kube-api-access-mx8sq\") pod \"maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8\" (UID: \"863ab859-100b-4fe9-843c-8390ebeab018\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8" Oct 30 21:45:00.575649 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.575608 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/154205a2-c374-4c7e-b869-c48c8f488db8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286\" (UID: \"154205a2-c374-4c7e-b869-c48c8f488db8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286" Oct 30 21:45:00.575888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.575858 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/863ab859-100b-4fe9-843c-8390ebeab018-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8\" (UID: \"863ab859-100b-4fe9-843c-8390ebeab018\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8" Oct 30 21:45:00.575955 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.575920 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b4wc\" (UniqueName: \"kubernetes.io/projected/154205a2-c374-4c7e-b869-c48c8f488db8-kube-api-access-4b4wc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286\" (UID: \"154205a2-c374-4c7e-b869-c48c8f488db8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286" Oct 30 21:45:00.576021 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.576006 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e76f09d3-4c42-4917-92cb-0f61d7cb381e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb\" (UID: \"e76f09d3-4c42-4917-92cb-0f61d7cb381e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb" Oct 30 21:45:00.576100 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.576088 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ad7312-8811-476b-9e22-2d062cfa5508-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp\" (UID: \"b4ad7312-8811-476b-9e22-2d062cfa5508\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp" Oct 30 21:45:00.576144 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.576134 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dbgs\" (UniqueName: \"kubernetes.io/projected/15e03e4b-deae-4513-83d2-efd7f5538d24-kube-api-access-5dbgs\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw\" (UID: \"15e03e4b-deae-4513-83d2-efd7f5538d24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw" Oct 30 21:45:00.576369 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.576350 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/863ab859-100b-4fe9-843c-8390ebeab018-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8\" (UID: \"863ab859-100b-4fe9-843c-8390ebeab018\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8" Oct 30 21:45:00.576720 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.576698 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ad7312-8811-476b-9e22-2d062cfa5508-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp\" (UID: \"b4ad7312-8811-476b-9e22-2d062cfa5508\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp" Oct 30 21:45:00.599362 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.599334 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5nzz\" (UniqueName: \"kubernetes.io/projected/e76f09d3-4c42-4917-92cb-0f61d7cb381e-kube-api-access-x5nzz\") pod \"maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb\" (UID: \"e76f09d3-4c42-4917-92cb-0f61d7cb381e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb" Oct 30 21:45:00.599877 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.599856 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx8sq\" (UniqueName: \"kubernetes.io/projected/863ab859-100b-4fe9-843c-8390ebeab018-kube-api-access-mx8sq\") pod \"maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8\" (UID: \"863ab859-100b-4fe9-843c-8390ebeab018\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8" Oct 30 21:45:00.601528 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.601502 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vkbb\" (UniqueName: \"kubernetes.io/projected/b4ad7312-8811-476b-9e22-2d062cfa5508-kube-api-access-5vkbb\") pod \"maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp\" (UID: \"b4ad7312-8811-476b-9e22-2d062cfa5508\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp" Oct 30 21:45:00.635405 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.635380 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9" Oct 30 21:45:00.664172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.664149 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52" Oct 30 21:45:00.672285 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.672264 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb" Oct 30 21:45:00.677555 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.677530 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/245e03b0-e13d-42fd-a32f-33891baa7f44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6\" (UID: \"245e03b0-e13d-42fd-a32f-33891baa7f44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6" Oct 30 21:45:00.677639 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.677628 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw8sm\" (UniqueName: \"kubernetes.io/projected/245e03b0-e13d-42fd-a32f-33891baa7f44-kube-api-access-kw8sm\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6\" (UID: \"245e03b0-e13d-42fd-a32f-33891baa7f44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6" Oct 30 21:45:00.677688 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.677673 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/154205a2-c374-4c7e-b869-c48c8f488db8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286\" (UID: \"154205a2-c374-4c7e-b869-c48c8f488db8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286" Oct 30 21:45:00.677739 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.677719 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b4wc\" (UniqueName: \"kubernetes.io/projected/154205a2-c374-4c7e-b869-c48c8f488db8-kube-api-access-4b4wc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286\" (UID: \"154205a2-c374-4c7e-b869-c48c8f488db8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286" Oct 30 21:45:00.678839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.677914 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dbgs\" (UniqueName: \"kubernetes.io/projected/15e03e4b-deae-4513-83d2-efd7f5538d24-kube-api-access-5dbgs\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw\" (UID: \"15e03e4b-deae-4513-83d2-efd7f5538d24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw" Oct 30 21:45:00.678839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.678113 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15e03e4b-deae-4513-83d2-efd7f5538d24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw\" (UID: \"15e03e4b-deae-4513-83d2-efd7f5538d24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw" Oct 30 21:45:00.678839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.678429 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/245e03b0-e13d-42fd-a32f-33891baa7f44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6\" (UID: \"245e03b0-e13d-42fd-a32f-33891baa7f44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6" Oct 30 21:45:00.678839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.678565 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/154205a2-c374-4c7e-b869-c48c8f488db8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286\" (UID: \"154205a2-c374-4c7e-b869-c48c8f488db8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286" Oct 30 21:45:00.678839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.678786 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15e03e4b-deae-4513-83d2-efd7f5538d24-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw\" (UID: \"15e03e4b-deae-4513-83d2-efd7f5538d24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw" Oct 30 21:45:00.679761 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.679728 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp" Oct 30 21:45:00.685327 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.684991 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8" Oct 30 21:45:00.719248 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.715010 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8" Oct 30 21:45:00.726177 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.725970 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b4wc\" (UniqueName: \"kubernetes.io/projected/154205a2-c374-4c7e-b869-c48c8f488db8-kube-api-access-4b4wc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286\" (UID: \"154205a2-c374-4c7e-b869-c48c8f488db8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286" Oct 30 21:45:00.728186 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.728010 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw8sm\" (UniqueName: \"kubernetes.io/projected/245e03b0-e13d-42fd-a32f-33891baa7f44-kube-api-access-kw8sm\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6\" (UID: \"245e03b0-e13d-42fd-a32f-33891baa7f44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6" Oct 30 21:45:00.730206 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.728928 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dbgs\" (UniqueName: \"kubernetes.io/projected/15e03e4b-deae-4513-83d2-efd7f5538d24-kube-api-access-5dbgs\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw\" (UID: \"15e03e4b-deae-4513-83d2-efd7f5538d24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw" Oct 30 21:45:00.741316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.741105 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw" Oct 30 21:45:00.749216 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.749016 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286" Oct 30 21:45:00.858343 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.858296 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:00.962711 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:45:00.961824 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fc9f9d_9a00_4001_b258_7f7850297a0e.slice/crio-5f07b8d9b0ac021b5221a19de9e1554d464503115f00fe8827c98f45daf3b169 WatchSource:0}: Error finding container 5f07b8d9b0ac021b5221a19de9e1554d464503115f00fe8827c98f45daf3b169: Status 404 returned error can't find the container with id 5f07b8d9b0ac021b5221a19de9e1554d464503115f00fe8827c98f45daf3b169 Oct 30 21:45:00.971244 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:00.971024 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9"] Oct 30 21:45:01.014841 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:01.014594 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6" Oct 30 21:45:01.066739 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:01.066705 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9" event={"ID":"51fc9f9d-9a00-4001-b258-7f7850297a0e","Type":"ContainerStarted","Data":"5f07b8d9b0ac021b5221a19de9e1554d464503115f00fe8827c98f45daf3b169"} Oct 30 21:45:01.182437 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:01.182369 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52"] Oct 30 21:45:01.203746 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:01.203719 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb"] Oct 30 21:45:01.256407 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:01.256365 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8"] Oct 30 21:45:01.266116 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:01.265378 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp"] Oct 30 21:45:01.324296 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:01.324266 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw"] Oct 30 21:45:01.328987 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:45:01.328943 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e03e4b_deae_4513_83d2_efd7f5538d24.slice/crio-77a429e5a824d955476c64eaa820fb857c96ac9cbe0621224ed98e7923e8220f WatchSource:0}: Error finding container 77a429e5a824d955476c64eaa820fb857c96ac9cbe0621224ed98e7923e8220f: Status 404 returned error can't find the container with id 77a429e5a824d955476c64eaa820fb857c96ac9cbe0621224ed98e7923e8220f Oct 30 21:45:01.333551 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:01.333526 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6"] Oct 30 21:45:01.349948 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:01.349918 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286"] Oct 30 21:45:01.351636 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:45:01.351612 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154205a2_c374_4c7e_b869_c48c8f488db8.slice/crio-55ec978bbbe0cb7873a451df967f82e7f2678b9f6b8b48f3469dc2d03ee44810 WatchSource:0}: Error finding container 55ec978bbbe0cb7873a451df967f82e7f2678b9f6b8b48f3469dc2d03ee44810: Status 404 returned error can't find the container with id 55ec978bbbe0cb7873a451df967f82e7f2678b9f6b8b48f3469dc2d03ee44810 Oct 30 21:45:01.375767 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:01.374792 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8"] Oct 30 21:45:01.791495 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:01.791464 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:02.079262 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:02.079223 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb" event={"ID":"e76f09d3-4c42-4917-92cb-0f61d7cb381e","Type":"ContainerStarted","Data":"719924a1dd09c79a43d2c825898d75de2ecaad759e4ba4a2f4bfaaae7bea2dc5"} Oct 30 21:45:02.079666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:02.079284 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb" event={"ID":"e76f09d3-4c42-4917-92cb-0f61d7cb381e","Type":"ContainerStarted","Data":"676eaecc7e14b9d5844bffecee6c04608c05875b8a5f293ae7fb5b3c9edd819d"} Oct 30 21:45:02.081253 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:02.081227 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52" event={"ID":"f58a71e2-3701-43d1-a341-58e5bef8732d","Type":"ContainerStarted","Data":"cd591fbdf2ef8330416f35e7c81b2eeecc38a6801ee51e975448fb07d15d29cd"} Oct 30 21:45:02.081390 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:02.081271 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52" event={"ID":"f58a71e2-3701-43d1-a341-58e5bef8732d","Type":"ContainerStarted","Data":"16384554279a17c69af2cf33856d42daf60bac619a8ba1d645926cf5f99155e5"} Oct 30 21:45:02.083113 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:02.083087 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9" event={"ID":"51fc9f9d-9a00-4001-b258-7f7850297a0e","Type":"ContainerStarted","Data":"5a10dd30cd27949f2df0a366426f6976b7bfb79623ebb75753a354c18efb8686"} Oct 30 21:45:02.084477 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:02.084454 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286" event={"ID":"154205a2-c374-4c7e-b869-c48c8f488db8","Type":"ContainerStarted","Data":"55ec978bbbe0cb7873a451df967f82e7f2678b9f6b8b48f3469dc2d03ee44810"} Oct 30 21:45:02.085686 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:02.085662 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw" event={"ID":"15e03e4b-deae-4513-83d2-efd7f5538d24","Type":"ContainerStarted","Data":"77a429e5a824d955476c64eaa820fb857c96ac9cbe0621224ed98e7923e8220f"} Oct 30 21:45:02.087101 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:02.087079 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8" event={"ID":"863ab859-100b-4fe9-843c-8390ebeab018","Type":"ContainerStarted","Data":"f3164346be13d5ee86bbe9fccff1e7d0c8fc1265ad94a9da982a0d516d4d50f2"} Oct 30 21:45:02.088708 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:02.088684 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp" event={"ID":"b4ad7312-8811-476b-9e22-2d062cfa5508","Type":"ContainerStarted","Data":"889f8c9de77ab377a6713ef5e1dec3e1c0cc14df0561b21d7d5e28f4b849eaf6"} Oct 30 21:45:02.088998 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:02.088732 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp" event={"ID":"b4ad7312-8811-476b-9e22-2d062cfa5508","Type":"ContainerStarted","Data":"9be5057949c7eeb98c90f3eed93c332ee9c8fd94ef731a5778c68c40bec8957c"} Oct 30 21:45:02.090372 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:02.090349 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8" event={"ID":"e4314f13-77fd-48d6-9387-3648850025da","Type":"ContainerStarted","Data":"d026a240d46c23b51736e8407bb12b5266384bf033eaeb37b5e37ee12721312c"} Oct 30 21:45:02.090475 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:02.090391 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8" event={"ID":"e4314f13-77fd-48d6-9387-3648850025da","Type":"ContainerStarted","Data":"bb56dd736b58786cc378c91df2b6fbeedb737dab1949f0a0fcbcd4b280919071"} Oct 30 21:45:02.091653 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:02.091634 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6" event={"ID":"245e03b0-e13d-42fd-a32f-33891baa7f44","Type":"ContainerStarted","Data":"46ef4d6bc752862fb164c39edbd07261b1037efc8e8a73c8ac1972edf79f9750"} Oct 30 21:45:02.100320 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:02.100280 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb" podStartSLOduration=2.100268785 podStartE2EDuration="2.100268785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.097771167 +0000 UTC m=+970.693587833" watchObservedRunningTime="2026-10-30 21:45:02.100268785 +0000 UTC m=+970.696085444" Oct 30 21:45:02.120574 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:02.120518 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8" podStartSLOduration=2.120504506 podStartE2EDuration="2.120504506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.115844814 +0000 UTC m=+970.711661479" watchObservedRunningTime="2026-10-30 21:45:02.120504506 +0000 UTC m=+970.716321166" Oct 30 21:45:02.134395 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:02.134350 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp" podStartSLOduration=2.134338488 podStartE2EDuration="2.134338488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.133597494 +0000 UTC m=+970.729414178" watchObservedRunningTime="2026-10-30 21:45:02.134338488 +0000 UTC m=+970.730155143" Oct 30 21:45:02.158471 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:02.158410 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9" podStartSLOduration=2.158391745 podStartE2EDuration="2.158391745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.156689242 +0000 UTC m=+970.752505901" watchObservedRunningTime="2026-10-30 21:45:02.158391745 +0000 UTC m=+970.754208407" Oct 30 21:45:02.181502 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:02.180294 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52" podStartSLOduration=2.180275633 podStartE2EDuration="2.180275633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.177348498 +0000 UTC m=+970.773165188" watchObservedRunningTime="2026-10-30 21:45:02.180275633 +0000 UTC m=+970.776092295" Oct 30 21:45:02.469212 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:02.468833 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:02.602397 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:02.602363 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:02.806622 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:02.806582 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:03.118120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.118087 5801 generic.go:356] "Generic (PLEG): container finished" podID="e76f09d3-4c42-4917-92cb-0f61d7cb381e" containerID="719924a1dd09c79a43d2c825898d75de2ecaad759e4ba4a2f4bfaaae7bea2dc5" exitCode=60 Oct 30 21:45:03.118548 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.118166 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb" event={"ID":"e76f09d3-4c42-4917-92cb-0f61d7cb381e","Type":"ContainerDied","Data":"719924a1dd09c79a43d2c825898d75de2ecaad759e4ba4a2f4bfaaae7bea2dc5"} Oct 30 21:45:03.118548 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.118481 5801 scope.go:122] "RemoveContainer" containerID="719924a1dd09c79a43d2c825898d75de2ecaad759e4ba4a2f4bfaaae7bea2dc5" Oct 30 21:45:03.120064 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.120036 5801 generic.go:356] "Generic (PLEG): container finished" podID="f58a71e2-3701-43d1-a341-58e5bef8732d" containerID="cd591fbdf2ef8330416f35e7c81b2eeecc38a6801ee51e975448fb07d15d29cd" exitCode=60 Oct 30 21:45:03.120199 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.120181 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52" event={"ID":"f58a71e2-3701-43d1-a341-58e5bef8732d","Type":"ContainerDied","Data":"cd591fbdf2ef8330416f35e7c81b2eeecc38a6801ee51e975448fb07d15d29cd"} Oct 30 21:45:03.120355 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.120340 5801 scope.go:122] "RemoveContainer" containerID="cd591fbdf2ef8330416f35e7c81b2eeecc38a6801ee51e975448fb07d15d29cd" Oct 30 21:45:03.122386 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.122304 5801 generic.go:356] "Generic (PLEG): container finished" podID="51fc9f9d-9a00-4001-b258-7f7850297a0e" containerID="5a10dd30cd27949f2df0a366426f6976b7bfb79623ebb75753a354c18efb8686" exitCode=60 Oct 30 21:45:03.122386 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.122327 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9" event={"ID":"51fc9f9d-9a00-4001-b258-7f7850297a0e","Type":"ContainerDied","Data":"5a10dd30cd27949f2df0a366426f6976b7bfb79623ebb75753a354c18efb8686"} Oct 30 21:45:03.122658 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.122634 5801 scope.go:122] "RemoveContainer" containerID="5a10dd30cd27949f2df0a366426f6976b7bfb79623ebb75753a354c18efb8686" Oct 30 21:45:03.124838 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.124819 5801 generic.go:356] "Generic (PLEG): container finished" podID="154205a2-c374-4c7e-b869-c48c8f488db8" containerID="7712f69d7e316a33926af8c6dffede7c010f0a384adddd9f50a69f92af7940d2" exitCode=60 Oct 30 21:45:03.124919 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.124904 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286" event={"ID":"154205a2-c374-4c7e-b869-c48c8f488db8","Type":"ContainerDied","Data":"7712f69d7e316a33926af8c6dffede7c010f0a384adddd9f50a69f92af7940d2"} Oct 30 21:45:03.125132 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.125116 5801 scope.go:122] "RemoveContainer" containerID="7712f69d7e316a33926af8c6dffede7c010f0a384adddd9f50a69f92af7940d2" Oct 30 21:45:03.126322 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.126303 5801 generic.go:356] "Generic (PLEG): container finished" podID="15e03e4b-deae-4513-83d2-efd7f5538d24" containerID="83985155f999914a06622abdee6726ef20dad2af5c9c19fea3f64131e0520be3" exitCode=60 Oct 30 21:45:03.126411 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.126338 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw" event={"ID":"15e03e4b-deae-4513-83d2-efd7f5538d24","Type":"ContainerDied","Data":"83985155f999914a06622abdee6726ef20dad2af5c9c19fea3f64131e0520be3"} Oct 30 21:45:03.126615 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.126598 5801 scope.go:122] "RemoveContainer" containerID="83985155f999914a06622abdee6726ef20dad2af5c9c19fea3f64131e0520be3" Oct 30 21:45:03.127873 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.127854 5801 generic.go:356] "Generic (PLEG): container finished" podID="863ab859-100b-4fe9-843c-8390ebeab018" containerID="0a0ac3c2aabb6ee15c87959f5961e1a41a9502feb4037768429cb560d2fd1602" exitCode=60 Oct 30 21:45:03.127958 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.127921 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8" event={"ID":"863ab859-100b-4fe9-843c-8390ebeab018","Type":"ContainerDied","Data":"0a0ac3c2aabb6ee15c87959f5961e1a41a9502feb4037768429cb560d2fd1602"} Oct 30 21:45:03.128137 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.128121 5801 scope.go:122] "RemoveContainer" containerID="0a0ac3c2aabb6ee15c87959f5961e1a41a9502feb4037768429cb560d2fd1602" Oct 30 21:45:03.129488 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.129468 5801 generic.go:356] "Generic (PLEG): container finished" podID="b4ad7312-8811-476b-9e22-2d062cfa5508" containerID="889f8c9de77ab377a6713ef5e1dec3e1c0cc14df0561b21d7d5e28f4b849eaf6" exitCode=60 Oct 30 21:45:03.129565 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.129548 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp" event={"ID":"b4ad7312-8811-476b-9e22-2d062cfa5508","Type":"ContainerDied","Data":"889f8c9de77ab377a6713ef5e1dec3e1c0cc14df0561b21d7d5e28f4b849eaf6"} Oct 30 21:45:03.129740 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.129723 5801 scope.go:122] "RemoveContainer" containerID="889f8c9de77ab377a6713ef5e1dec3e1c0cc14df0561b21d7d5e28f4b849eaf6" Oct 30 21:45:03.131253 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.131222 5801 generic.go:356] "Generic (PLEG): container finished" podID="e4314f13-77fd-48d6-9387-3648850025da" containerID="d026a240d46c23b51736e8407bb12b5266384bf033eaeb37b5e37ee12721312c" exitCode=60 Oct 30 21:45:03.131341 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.131320 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8" event={"ID":"e4314f13-77fd-48d6-9387-3648850025da","Type":"ContainerDied","Data":"d026a240d46c23b51736e8407bb12b5266384bf033eaeb37b5e37ee12721312c"} Oct 30 21:45:03.131541 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.131522 5801 scope.go:122] "RemoveContainer" containerID="d026a240d46c23b51736e8407bb12b5266384bf033eaeb37b5e37ee12721312c" Oct 30 21:45:03.135230 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.135210 5801 generic.go:356] "Generic (PLEG): container finished" podID="245e03b0-e13d-42fd-a32f-33891baa7f44" containerID="97aa4fd9a8f48ed064f6da378f2f0dd984e3bd595d06cd5c4c81fec84ed87646" exitCode=60 Oct 30 21:45:03.135321 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.135235 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6" event={"ID":"245e03b0-e13d-42fd-a32f-33891baa7f44","Type":"ContainerDied","Data":"97aa4fd9a8f48ed064f6da378f2f0dd984e3bd595d06cd5c4c81fec84ed87646"} Oct 30 21:45:03.135484 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.135471 5801 scope.go:122] "RemoveContainer" containerID="97aa4fd9a8f48ed064f6da378f2f0dd984e3bd595d06cd5c4c81fec84ed87646" Oct 30 21:45:03.697628 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.697558 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:03.806905 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:03.806870 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:04.186911 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:04.186870 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb" event={"ID":"e76f09d3-4c42-4917-92cb-0f61d7cb381e","Type":"ContainerStarted","Data":"4cba48dc748e474d599b1b89c28c412bb1922b5b0b92177b65917e781c0ed229"} Oct 30 21:45:04.189681 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:04.189654 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52" event={"ID":"f58a71e2-3701-43d1-a341-58e5bef8732d","Type":"ContainerStarted","Data":"89993cccd296ca5efb1916f58cbd601d75346b4fba819ae6b3560ff4b645eb9e"} Oct 30 21:45:04.193260 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:04.193232 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9" event={"ID":"51fc9f9d-9a00-4001-b258-7f7850297a0e","Type":"ContainerStarted","Data":"b07c027b043693617bbe3861271fc7c66269fe1329acf8a303db77a997f4da18"} Oct 30 21:45:04.196629 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:04.196564 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286" event={"ID":"154205a2-c374-4c7e-b869-c48c8f488db8","Type":"ContainerStarted","Data":"4753b9d11c14c5dd77f0d35d17ba18aecc3fe3f670fe62ee72f9e52593fe78a6"} Oct 30 21:45:04.199022 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:04.198999 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw" event={"ID":"15e03e4b-deae-4513-83d2-efd7f5538d24","Type":"ContainerStarted","Data":"a9078c508e3330493b7b170368f35cb783fc5819ff42f4d991e309172ffd73e2"} Oct 30 21:45:04.201041 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:04.200958 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8" event={"ID":"863ab859-100b-4fe9-843c-8390ebeab018","Type":"ContainerStarted","Data":"0a306a44d4fd12d5b3f69518368c6d721b818a4cfa62fa9f376b7c52be1b29cd"} Oct 30 21:45:04.203389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:04.203363 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp" event={"ID":"b4ad7312-8811-476b-9e22-2d062cfa5508","Type":"ContainerStarted","Data":"2502711202a68e51106083f2ed1708b5a4a50a07c22d7691afd46aa2ac80708e"} Oct 30 21:45:04.205577 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:04.205555 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8" event={"ID":"e4314f13-77fd-48d6-9387-3648850025da","Type":"ContainerStarted","Data":"ad3e544fe1e02bc0475f8df9948ea4c6ab94e71ff1a846a89796f5406efa5ee8"} Oct 30 21:45:04.207714 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:04.207691 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6" event={"ID":"245e03b0-e13d-42fd-a32f-33891baa7f44","Type":"ContainerStarted","Data":"c638dc71787df96e67a89b24d3e58d86f87d5d5f2b350f6f8b5b1a50c1dfd10b"} Oct 30 21:45:04.382894 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:04.382836 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8" podStartSLOduration=4.382816757 podStartE2EDuration="4.382816757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.355997768 +0000 UTC m=+972.951814429" watchObservedRunningTime="2026-10-30 21:45:04.382816757 +0000 UTC m=+972.978633420" Oct 30 21:45:04.399060 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:04.399015 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6" podStartSLOduration=4.399002663 podStartE2EDuration="4.399002663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.396697328 +0000 UTC m=+972.992513987" watchObservedRunningTime="2026-10-30 21:45:04.399002663 +0000 UTC m=+972.994819324" Oct 30 21:45:04.418458 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:04.418416 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw" podStartSLOduration=4.418404573 podStartE2EDuration="4.418404573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.418000938 +0000 UTC m=+973.013817598" watchObservedRunningTime="2026-10-30 21:45:04.418404573 +0000 UTC m=+973.014221233" Oct 30 21:45:04.422199 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:04.422177 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:04.440688 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:04.440608 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286" podStartSLOduration=4.440595676 podStartE2EDuration="4.440595676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.436882039 +0000 UTC m=+973.032698710" watchObservedRunningTime="2026-10-30 21:45:04.440595676 +0000 UTC m=+973.036412336" Oct 30 21:45:05.294810 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.294771 5801 generic.go:356] "Generic (PLEG): container finished" podID="e4314f13-77fd-48d6-9387-3648850025da" containerID="ad3e544fe1e02bc0475f8df9948ea4c6ab94e71ff1a846a89796f5406efa5ee8" exitCode=60 Oct 30 21:45:05.295444 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.295417 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8" event={"ID":"e4314f13-77fd-48d6-9387-3648850025da","Type":"ContainerDied","Data":"ad3e544fe1e02bc0475f8df9948ea4c6ab94e71ff1a846a89796f5406efa5ee8"} Oct 30 21:45:05.295635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.295620 5801 scope.go:122] "RemoveContainer" containerID="d026a240d46c23b51736e8407bb12b5266384bf033eaeb37b5e37ee12721312c" Oct 30 21:45:05.296326 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.296306 5801 scope.go:122] "RemoveContainer" containerID="ad3e544fe1e02bc0475f8df9948ea4c6ab94e71ff1a846a89796f5406efa5ee8" Oct 30 21:45:05.296958 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:05.296932 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8_odh-ai-gateway-infra(e4314f13-77fd-48d6-9387-3648850025da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8" podUID="e4314f13-77fd-48d6-9387-3648850025da" Oct 30 21:45:05.329785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.329367 5801 generic.go:356] "Generic (PLEG): container finished" podID="245e03b0-e13d-42fd-a32f-33891baa7f44" containerID="c638dc71787df96e67a89b24d3e58d86f87d5d5f2b350f6f8b5b1a50c1dfd10b" exitCode=60 Oct 30 21:45:05.329785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.329740 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6" event={"ID":"245e03b0-e13d-42fd-a32f-33891baa7f44","Type":"ContainerDied","Data":"c638dc71787df96e67a89b24d3e58d86f87d5d5f2b350f6f8b5b1a50c1dfd10b"} Oct 30 21:45:05.374806 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.368123 5801 scope.go:122] "RemoveContainer" containerID="c638dc71787df96e67a89b24d3e58d86f87d5d5f2b350f6f8b5b1a50c1dfd10b" Oct 30 21:45:05.374806 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:05.368639 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6_odh-ai-gateway-infra(245e03b0-e13d-42fd-a32f-33891baa7f44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6" podUID="245e03b0-e13d-42fd-a32f-33891baa7f44" Oct 30 21:45:05.385843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.385729 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:05.396415 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.389656 5801 generic.go:356] "Generic (PLEG): container finished" podID="e76f09d3-4c42-4917-92cb-0f61d7cb381e" containerID="4cba48dc748e474d599b1b89c28c412bb1922b5b0b92177b65917e781c0ed229" exitCode=60 Oct 30 21:45:05.396415 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.389788 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb" event={"ID":"e76f09d3-4c42-4917-92cb-0f61d7cb381e","Type":"ContainerDied","Data":"4cba48dc748e474d599b1b89c28c412bb1922b5b0b92177b65917e781c0ed229"} Oct 30 21:45:05.396415 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.390325 5801 scope.go:122] "RemoveContainer" containerID="4cba48dc748e474d599b1b89c28c412bb1922b5b0b92177b65917e781c0ed229" Oct 30 21:45:05.399036 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:05.397879 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb_odh-ai-gateway-infra(e76f09d3-4c42-4917-92cb-0f61d7cb381e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb" podUID="e76f09d3-4c42-4917-92cb-0f61d7cb381e" Oct 30 21:45:05.468827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.468797 5801 scope.go:122] "RemoveContainer" containerID="97aa4fd9a8f48ed064f6da378f2f0dd984e3bd595d06cd5c4c81fec84ed87646" Oct 30 21:45:05.474379 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.473613 5801 generic.go:356] "Generic (PLEG): container finished" podID="f58a71e2-3701-43d1-a341-58e5bef8732d" containerID="89993cccd296ca5efb1916f58cbd601d75346b4fba819ae6b3560ff4b645eb9e" exitCode=60 Oct 30 21:45:05.474379 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.473874 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52" event={"ID":"f58a71e2-3701-43d1-a341-58e5bef8732d","Type":"ContainerDied","Data":"89993cccd296ca5efb1916f58cbd601d75346b4fba819ae6b3560ff4b645eb9e"} Oct 30 21:45:05.474379 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.474095 5801 scope.go:122] "RemoveContainer" containerID="89993cccd296ca5efb1916f58cbd601d75346b4fba819ae6b3560ff4b645eb9e" Oct 30 21:45:05.474597 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:05.474561 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52_odh-ai-gateway-infra(f58a71e2-3701-43d1-a341-58e5bef8732d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52" podUID="f58a71e2-3701-43d1-a341-58e5bef8732d" Oct 30 21:45:05.477284 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.477256 5801 generic.go:356] "Generic (PLEG): container finished" podID="51fc9f9d-9a00-4001-b258-7f7850297a0e" containerID="b07c027b043693617bbe3861271fc7c66269fe1329acf8a303db77a997f4da18" exitCode=60 Oct 30 21:45:05.477450 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.477369 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9" event={"ID":"51fc9f9d-9a00-4001-b258-7f7850297a0e","Type":"ContainerDied","Data":"b07c027b043693617bbe3861271fc7c66269fe1329acf8a303db77a997f4da18"} Oct 30 21:45:05.477638 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.477620 5801 scope.go:122] "RemoveContainer" containerID="b07c027b043693617bbe3861271fc7c66269fe1329acf8a303db77a997f4da18" Oct 30 21:45:05.478087 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:05.478064 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9_odh-ai-gateway-infra(51fc9f9d-9a00-4001-b258-7f7850297a0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9" podUID="51fc9f9d-9a00-4001-b258-7f7850297a0e" Oct 30 21:45:05.480446 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.480425 5801 generic.go:356] "Generic (PLEG): container finished" podID="154205a2-c374-4c7e-b869-c48c8f488db8" containerID="4753b9d11c14c5dd77f0d35d17ba18aecc3fe3f670fe62ee72f9e52593fe78a6" exitCode=60 Oct 30 21:45:05.480655 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.480632 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286" event={"ID":"154205a2-c374-4c7e-b869-c48c8f488db8","Type":"ContainerDied","Data":"4753b9d11c14c5dd77f0d35d17ba18aecc3fe3f670fe62ee72f9e52593fe78a6"} Oct 30 21:45:05.480918 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.480877 5801 scope.go:122] "RemoveContainer" containerID="4753b9d11c14c5dd77f0d35d17ba18aecc3fe3f670fe62ee72f9e52593fe78a6" Oct 30 21:45:05.481293 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:05.481266 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286_odh-ai-gateway-infra(154205a2-c374-4c7e-b869-c48c8f488db8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286" podUID="154205a2-c374-4c7e-b869-c48c8f488db8" Oct 30 21:45:05.485298 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.485276 5801 generic.go:356] "Generic (PLEG): container finished" podID="15e03e4b-deae-4513-83d2-efd7f5538d24" containerID="a9078c508e3330493b7b170368f35cb783fc5819ff42f4d991e309172ffd73e2" exitCode=60 Oct 30 21:45:05.485452 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.485348 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw" event={"ID":"15e03e4b-deae-4513-83d2-efd7f5538d24","Type":"ContainerDied","Data":"a9078c508e3330493b7b170368f35cb783fc5819ff42f4d991e309172ffd73e2"} Oct 30 21:45:05.485699 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.485678 5801 scope.go:122] "RemoveContainer" containerID="a9078c508e3330493b7b170368f35cb783fc5819ff42f4d991e309172ffd73e2" Oct 30 21:45:05.486518 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:05.486331 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw_odh-ai-gateway-infra(15e03e4b-deae-4513-83d2-efd7f5538d24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw" podUID="15e03e4b-deae-4513-83d2-efd7f5538d24" Oct 30 21:45:05.487779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.487730 5801 generic.go:356] "Generic (PLEG): container finished" podID="863ab859-100b-4fe9-843c-8390ebeab018" containerID="0a306a44d4fd12d5b3f69518368c6d721b818a4cfa62fa9f376b7c52be1b29cd" exitCode=60 Oct 30 21:45:05.487900 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.487877 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8" event={"ID":"863ab859-100b-4fe9-843c-8390ebeab018","Type":"ContainerDied","Data":"0a306a44d4fd12d5b3f69518368c6d721b818a4cfa62fa9f376b7c52be1b29cd"} Oct 30 21:45:05.488165 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.488147 5801 scope.go:122] "RemoveContainer" containerID="0a306a44d4fd12d5b3f69518368c6d721b818a4cfa62fa9f376b7c52be1b29cd" Oct 30 21:45:05.488517 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:05.488495 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8_odh-ai-gateway-infra(863ab859-100b-4fe9-843c-8390ebeab018)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8" podUID="863ab859-100b-4fe9-843c-8390ebeab018" Oct 30 21:45:05.490100 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.490080 5801 generic.go:356] "Generic (PLEG): container finished" podID="b4ad7312-8811-476b-9e22-2d062cfa5508" containerID="2502711202a68e51106083f2ed1708b5a4a50a07c22d7691afd46aa2ac80708e" exitCode=60 Oct 30 21:45:05.490199 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.490153 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp" event={"ID":"b4ad7312-8811-476b-9e22-2d062cfa5508","Type":"ContainerDied","Data":"2502711202a68e51106083f2ed1708b5a4a50a07c22d7691afd46aa2ac80708e"} Oct 30 21:45:05.490494 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.490477 5801 scope.go:122] "RemoveContainer" containerID="2502711202a68e51106083f2ed1708b5a4a50a07c22d7691afd46aa2ac80708e" Oct 30 21:45:05.490900 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:05.490846 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp_odh-ai-gateway-infra(b4ad7312-8811-476b-9e22-2d062cfa5508)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp" podUID="b4ad7312-8811-476b-9e22-2d062cfa5508" Oct 30 21:45:05.491567 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.491547 5801 scope.go:122] "RemoveContainer" containerID="719924a1dd09c79a43d2c825898d75de2ecaad759e4ba4a2f4bfaaae7bea2dc5" Oct 30 21:45:05.513850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.513823 5801 scope.go:122] "RemoveContainer" containerID="cd591fbdf2ef8330416f35e7c81b2eeecc38a6801ee51e975448fb07d15d29cd" Oct 30 21:45:05.531673 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.531656 5801 scope.go:122] "RemoveContainer" containerID="5a10dd30cd27949f2df0a366426f6976b7bfb79623ebb75753a354c18efb8686" Oct 30 21:45:05.556370 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.556348 5801 scope.go:122] "RemoveContainer" containerID="7712f69d7e316a33926af8c6dffede7c010f0a384adddd9f50a69f92af7940d2" Oct 30 21:45:05.576711 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.576689 5801 scope.go:122] "RemoveContainer" containerID="83985155f999914a06622abdee6726ef20dad2af5c9c19fea3f64131e0520be3" Oct 30 21:45:05.598255 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.598231 5801 scope.go:122] "RemoveContainer" containerID="0a0ac3c2aabb6ee15c87959f5961e1a41a9502feb4037768429cb560d2fd1602" Oct 30 21:45:05.614740 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:05.614717 5801 scope.go:122] "RemoveContainer" containerID="889f8c9de77ab377a6713ef5e1dec3e1c0cc14df0561b21d7d5e28f4b849eaf6" Oct 30 21:45:06.237120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:06.237072 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:06.511840 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:06.511707 5801 scope.go:122] "RemoveContainer" containerID="b07c027b043693617bbe3861271fc7c66269fe1329acf8a303db77a997f4da18" Oct 30 21:45:06.512213 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:06.512085 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9_odh-ai-gateway-infra(51fc9f9d-9a00-4001-b258-7f7850297a0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9" podUID="51fc9f9d-9a00-4001-b258-7f7850297a0e" Oct 30 21:45:06.513901 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:06.513878 5801 scope.go:122] "RemoveContainer" containerID="4753b9d11c14c5dd77f0d35d17ba18aecc3fe3f670fe62ee72f9e52593fe78a6" Oct 30 21:45:06.514176 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:06.514153 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286_odh-ai-gateway-infra(154205a2-c374-4c7e-b869-c48c8f488db8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286" podUID="154205a2-c374-4c7e-b869-c48c8f488db8" Oct 30 21:45:06.515618 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:06.515593 5801 scope.go:122] "RemoveContainer" containerID="a9078c508e3330493b7b170368f35cb783fc5819ff42f4d991e309172ffd73e2" Oct 30 21:45:06.515893 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:06.515875 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw_odh-ai-gateway-infra(15e03e4b-deae-4513-83d2-efd7f5538d24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw" podUID="15e03e4b-deae-4513-83d2-efd7f5538d24" Oct 30 21:45:06.517388 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:06.517367 5801 scope.go:122] "RemoveContainer" containerID="0a306a44d4fd12d5b3f69518368c6d721b818a4cfa62fa9f376b7c52be1b29cd" Oct 30 21:45:06.517643 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:06.517620 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8_odh-ai-gateway-infra(863ab859-100b-4fe9-843c-8390ebeab018)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8" podUID="863ab859-100b-4fe9-843c-8390ebeab018" Oct 30 21:45:06.519035 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:06.519019 5801 scope.go:122] "RemoveContainer" containerID="2502711202a68e51106083f2ed1708b5a4a50a07c22d7691afd46aa2ac80708e" Oct 30 21:45:06.519266 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:06.519248 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp_odh-ai-gateway-infra(b4ad7312-8811-476b-9e22-2d062cfa5508)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp" podUID="b4ad7312-8811-476b-9e22-2d062cfa5508" Oct 30 21:45:06.520567 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:06.520552 5801 scope.go:122] "RemoveContainer" containerID="ad3e544fe1e02bc0475f8df9948ea4c6ab94e71ff1a846a89796f5406efa5ee8" Oct 30 21:45:06.520832 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:06.520809 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8_odh-ai-gateway-infra(e4314f13-77fd-48d6-9387-3648850025da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8" podUID="e4314f13-77fd-48d6-9387-3648850025da" Oct 30 21:45:06.522141 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:06.522123 5801 scope.go:122] "RemoveContainer" containerID="c638dc71787df96e67a89b24d3e58d86f87d5d5f2b350f6f8b5b1a50c1dfd10b" Oct 30 21:45:06.522392 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:06.522373 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6_odh-ai-gateway-infra(245e03b0-e13d-42fd-a32f-33891baa7f44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6" podUID="245e03b0-e13d-42fd-a32f-33891baa7f44" Oct 30 21:45:06.523816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:06.523797 5801 scope.go:122] "RemoveContainer" containerID="4cba48dc748e474d599b1b89c28c412bb1922b5b0b92177b65917e781c0ed229" Oct 30 21:45:06.524040 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:06.524023 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb_odh-ai-gateway-infra(e76f09d3-4c42-4917-92cb-0f61d7cb381e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb" podUID="e76f09d3-4c42-4917-92cb-0f61d7cb381e" Oct 30 21:45:06.525356 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:06.525339 5801 scope.go:122] "RemoveContainer" containerID="89993cccd296ca5efb1916f58cbd601d75346b4fba819ae6b3560ff4b645eb9e" Oct 30 21:45:06.525544 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:06.525529 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52_odh-ai-gateway-infra(f58a71e2-3701-43d1-a341-58e5bef8732d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52" podUID="f58a71e2-3701-43d1-a341-58e5bef8732d" Oct 30 21:45:06.937034 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:06.936999 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:07.317328 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:07.317297 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:07.634183 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:07.634101 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:07.731727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:07.731697 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:07.839368 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:07.839238 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:08.453842 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:08.453803 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:09.034912 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:09.034885 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5" Oct 30 21:45:09.035296 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:09.034954 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k" Oct 30 21:45:09.269034 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:09.268992 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:10.251321 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:10.251286 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:10.692696 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:10.692656 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:11.059621 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:11.059586 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:11.751358 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:11.751299 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:12.476066 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:12.476025 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:13.167215 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:13.167175 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:14.044549 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:14.044515 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:14.146091 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:14.146058 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:14.442549 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:14.435831 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:14.552803 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:14.535291 5801 scope.go:122] "RemoveContainer" containerID="89993cccd296ca5efb1916f58cbd601d75346b4fba819ae6b3560ff4b645eb9e" Oct 30 21:45:14.976669 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:14.976636 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:15.520832 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:15.519385 5801 scope.go:122] "RemoveContainer" containerID="a9078c508e3330493b7b170368f35cb783fc5819ff42f4d991e309172ffd73e2" Oct 30 21:45:15.520832 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:15.520467 5801 scope.go:122] "RemoveContainer" containerID="2502711202a68e51106083f2ed1708b5a4a50a07c22d7691afd46aa2ac80708e" Oct 30 21:45:15.577812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:15.572094 5801 scope.go:122] "RemoveContainer" containerID="0a306a44d4fd12d5b3f69518368c6d721b818a4cfa62fa9f376b7c52be1b29cd" Oct 30 21:45:15.805316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:15.805283 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52" event={"ID":"f58a71e2-3701-43d1-a341-58e5bef8732d","Type":"ContainerStarted","Data":"511257441a4b0b450847060919134b941dd604b62765dadbd21df6a2b368631c"} Oct 30 21:45:15.947010 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:15.946978 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:16.517735 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:16.517712 5801 scope.go:122] "RemoveContainer" containerID="ad3e544fe1e02bc0475f8df9948ea4c6ab94e71ff1a846a89796f5406efa5ee8" Oct 30 21:45:16.518917 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:16.518899 5801 scope.go:122] "RemoveContainer" containerID="4753b9d11c14c5dd77f0d35d17ba18aecc3fe3f670fe62ee72f9e52593fe78a6" Oct 30 21:45:16.839443 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:16.838693 5801 generic.go:356] "Generic (PLEG): container finished" podID="f58a71e2-3701-43d1-a341-58e5bef8732d" containerID="511257441a4b0b450847060919134b941dd604b62765dadbd21df6a2b368631c" exitCode=60 Oct 30 21:45:16.839443 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:16.838886 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52" event={"ID":"f58a71e2-3701-43d1-a341-58e5bef8732d","Type":"ContainerDied","Data":"511257441a4b0b450847060919134b941dd604b62765dadbd21df6a2b368631c"} Oct 30 21:45:16.839443 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:16.838932 5801 scope.go:122] "RemoveContainer" containerID="89993cccd296ca5efb1916f58cbd601d75346b4fba819ae6b3560ff4b645eb9e" Oct 30 21:45:16.841173 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:16.840347 5801 scope.go:122] "RemoveContainer" containerID="511257441a4b0b450847060919134b941dd604b62765dadbd21df6a2b368631c" Oct 30 21:45:16.841173 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:16.840839 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52_odh-ai-gateway-infra(f58a71e2-3701-43d1-a341-58e5bef8732d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52" podUID="f58a71e2-3701-43d1-a341-58e5bef8732d" Oct 30 21:45:16.842827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:16.842805 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw" event={"ID":"15e03e4b-deae-4513-83d2-efd7f5538d24","Type":"ContainerStarted","Data":"5cec1a5ecb450a8e1d12a75d45b948bba1c5244f8a108618ac2def22359e8dc7"} Oct 30 21:45:16.844949 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:16.844923 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8" event={"ID":"863ab859-100b-4fe9-843c-8390ebeab018","Type":"ContainerStarted","Data":"3d11c823d2d984142c51a58b9f63f8d2173e753d92d877e8d0d5d51474076801"} Oct 30 21:45:16.849974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:16.849308 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp" event={"ID":"b4ad7312-8811-476b-9e22-2d062cfa5508","Type":"ContainerStarted","Data":"8acc4ff11450ad2671ab19c7bebfbf7392ac2d8a7bbd3e2e34653ee91f173d29"} Oct 30 21:45:16.849974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:16.849361 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52"] Oct 30 21:45:16.953783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:16.950156 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:17.532824 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:17.532729 5801 scope.go:122] "RemoveContainer" containerID="c638dc71787df96e67a89b24d3e58d86f87d5d5f2b350f6f8b5b1a50c1dfd10b" Oct 30 21:45:17.533716 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:17.533692 5801 scope.go:122] "RemoveContainer" containerID="b07c027b043693617bbe3861271fc7c66269fe1329acf8a303db77a997f4da18" Oct 30 21:45:17.534414 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:17.534395 5801 scope.go:122] "RemoveContainer" containerID="4cba48dc748e474d599b1b89c28c412bb1922b5b0b92177b65917e781c0ed229" Oct 30 21:45:17.737356 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:17.737323 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:18.250020 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:18.249989 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp"] Oct 30 21:45:18.657797 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:18.656596 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:18.813826 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:18.802921 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:18.847115 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:18.838766 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8"] Oct 30 21:45:19.051791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:18.989628 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:19.051791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.021806 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:45:19.167809 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.157043 5801 generic.go:356] "Generic (PLEG): container finished" podID="154205a2-c374-4c7e-b869-c48c8f488db8" containerID="c6b1e9094fa85c41f02da649311cc0afdddfbd432bc0bab89a54c1115cb857f0" exitCode=60 Oct 30 21:45:19.167809 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.157206 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286" event={"ID":"154205a2-c374-4c7e-b869-c48c8f488db8","Type":"ContainerDied","Data":"c6b1e9094fa85c41f02da649311cc0afdddfbd432bc0bab89a54c1115cb857f0"} Oct 30 21:45:19.167809 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.157268 5801 scope.go:122] "RemoveContainer" containerID="4753b9d11c14c5dd77f0d35d17ba18aecc3fe3f670fe62ee72f9e52593fe78a6" Oct 30 21:45:19.167809 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.157870 5801 scope.go:122] "RemoveContainer" containerID="c6b1e9094fa85c41f02da649311cc0afdddfbd432bc0bab89a54c1115cb857f0" Oct 30 21:45:19.167809 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:19.158360 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286_odh-ai-gateway-infra(154205a2-c374-4c7e-b869-c48c8f488db8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286" podUID="154205a2-c374-4c7e-b869-c48c8f488db8" Oct 30 21:45:19.167809 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.159109 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:19.188613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.174714 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:19.188613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.175082 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:45:19.188613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.175349 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:45:19.188613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.176832 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:45:19.227932 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.225908 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:19.251638 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.250842 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:19.283775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.282980 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:19.283775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.283337 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:19.283775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.283606 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Oct 30 21:45:19.295642 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.285645 5801 generic.go:356] "Generic (PLEG): container finished" podID="15e03e4b-deae-4513-83d2-efd7f5538d24" containerID="5cec1a5ecb450a8e1d12a75d45b948bba1c5244f8a108618ac2def22359e8dc7" exitCode=60 Oct 30 21:45:19.295642 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.285734 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw" event={"ID":"15e03e4b-deae-4513-83d2-efd7f5538d24","Type":"ContainerDied","Data":"5cec1a5ecb450a8e1d12a75d45b948bba1c5244f8a108618ac2def22359e8dc7"} Oct 30 21:45:19.295642 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.286253 5801 scope.go:122] "RemoveContainer" containerID="5cec1a5ecb450a8e1d12a75d45b948bba1c5244f8a108618ac2def22359e8dc7" Oct 30 21:45:19.295642 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:19.286722 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw_odh-ai-gateway-infra(15e03e4b-deae-4513-83d2-efd7f5538d24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw" podUID="15e03e4b-deae-4513-83d2-efd7f5538d24" Oct 30 21:45:19.600235 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.592840 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:45:19.600235 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.593907 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:19.701856 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.670435 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:45:19.701856 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.686363 5801 generic.go:356] "Generic (PLEG): container finished" podID="863ab859-100b-4fe9-843c-8390ebeab018" containerID="3d11c823d2d984142c51a58b9f63f8d2173e753d92d877e8d0d5d51474076801" exitCode=60 Oct 30 21:45:19.752549 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.752450 5801 scope.go:122] "RemoveContainer" containerID="a9078c508e3330493b7b170368f35cb783fc5819ff42f4d991e309172ffd73e2" Oct 30 21:45:19.765245 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.755604 5801 generic.go:356] "Generic (PLEG): container finished" podID="b4ad7312-8811-476b-9e22-2d062cfa5508" containerID="8acc4ff11450ad2671ab19c7bebfbf7392ac2d8a7bbd3e2e34653ee91f173d29" exitCode=60 Oct 30 21:45:19.862933 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.852031 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-448hk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:19.862933 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.852158 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" podUID="79d7a2c8-0585-48a3-88d4-d49b23672216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:19.964305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:19.964266 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:20.161845 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:20.085415 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:45:20.200907 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:20.165107 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:20.352125 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:20.181847 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:20.397829 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:20.181894 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:45:20.452371 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:20.181915 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:45:20.476419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:20.181934 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:45:20.477176 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:20.286321 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:20.499882 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:20.286623 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:45:20.500611 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:20.295550 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:45:20.501111 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:20.402256 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 21:45:20.730430 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:20.703952 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:45:20.730430 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:20.718034 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:20.731114 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:20.731082 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:20.906622 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:20.906580 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:20.962711 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:20.962676 5801 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.325s" Oct 30 21:45:20.963044 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:20.963026 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:45:20.963196 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:20.963181 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw"] Oct 30 21:45:20.963367 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:20.963339 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8" event={"ID":"863ab859-100b-4fe9-843c-8390ebeab018","Type":"ContainerDied","Data":"3d11c823d2d984142c51a58b9f63f8d2173e753d92d877e8d0d5d51474076801"} Oct 30 21:45:20.963597 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:20.963582 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:45:20.963721 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:20.963706 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp" event={"ID":"b4ad7312-8811-476b-9e22-2d062cfa5508","Type":"ContainerDied","Data":"8acc4ff11450ad2671ab19c7bebfbf7392ac2d8a7bbd3e2e34653ee91f173d29"} Oct 30 21:45:20.963904 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:20.963871 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:45:20.964061 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:20.964048 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:45:21.026821 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.026735 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:45:21.027028 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.027006 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8" event={"ID":"e4314f13-77fd-48d6-9387-3648850025da","Type":"ContainerStarted","Data":"3b0fe246887db49a3fba184e07b776fb8ba8bb6176fa12cefd4b862219e8a732"} Oct 30 21:45:21.027184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.027171 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:45:21.027297 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.027287 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:45:21.027408 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.027399 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:45:21.027510 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.027501 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:45:21.027610 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.027602 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:45:21.027718 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.027709 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:45:21.027932 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.027918 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:45:21.028049 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.028039 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:45:21.054777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.040489 5801 scope.go:122] "RemoveContainer" containerID="3b0fe246887db49a3fba184e07b776fb8ba8bb6176fa12cefd4b862219e8a732" Oct 30 21:45:21.247868 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:21.246907 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8_odh-ai-gateway-infra(e4314f13-77fd-48d6-9387-3648850025da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8" podUID="e4314f13-77fd-48d6-9387-3648850025da" Oct 30 21:45:21.247868 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.236088 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded" Oct 30 21:45:21.247868 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.098691 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:45:21.248389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.248366 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:45:21.259120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.258614 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:45:21.269667 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.267021 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:45:21.279158 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.279090 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:45:21.281371 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.281348 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:45:21.281472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.281392 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:45:21.281472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.281422 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:45:21.326039 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.326009 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286"] Oct 30 21:45:21.433984 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.433941 5801 scope.go:122] "RemoveContainer" containerID="0a306a44d4fd12d5b3f69518368c6d721b818a4cfa62fa9f376b7c52be1b29cd" Oct 30 21:45:21.443449 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.443419 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:45:21.443901 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.443513 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:45:21.447724 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.447705 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:45:21.518250 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:21.518205 5801 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245e03b0_e13d_42fd_a32f_33891baa7f44.slice/crio-conmon-058aeada96f7aba93c3c83d4a4c37da1f164abad83aff5d9da176c3a859badfb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245e03b0_e13d_42fd_a32f_33891baa7f44.slice/crio-058aeada96f7aba93c3c83d4a4c37da1f164abad83aff5d9da176c3a859badfb.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:45:21.523564 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.522054 5801 generic.go:356] "Generic (PLEG): container finished" podID="e4314f13-77fd-48d6-9387-3648850025da" containerID="3b0fe246887db49a3fba184e07b776fb8ba8bb6176fa12cefd4b862219e8a732" exitCode=60 Oct 30 21:45:21.542808 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:21.529860 5801 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245e03b0_e13d_42fd_a32f_33891baa7f44.slice/crio-conmon-058aeada96f7aba93c3c83d4a4c37da1f164abad83aff5d9da176c3a859badfb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245e03b0_e13d_42fd_a32f_33891baa7f44.slice/crio-058aeada96f7aba93c3c83d4a4c37da1f164abad83aff5d9da176c3a859badfb.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:45:21.542808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.540954 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:45:21.542808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.541000 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8" event={"ID":"e4314f13-77fd-48d6-9387-3648850025da","Type":"ContainerDied","Data":"3b0fe246887db49a3fba184e07b776fb8ba8bb6176fa12cefd4b862219e8a732"} Oct 30 21:45:21.564745 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.564214 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:45:21.590136 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.590025 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52" Oct 30 21:45:21.622769 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.611560 5801 scope.go:122] "RemoveContainer" containerID="2502711202a68e51106083f2ed1708b5a4a50a07c22d7691afd46aa2ac80708e" Oct 30 21:45:21.637877 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.637844 5801 generic.go:356] "Generic (PLEG): container finished" podID="245e03b0-e13d-42fd-a32f-33891baa7f44" containerID="058aeada96f7aba93c3c83d4a4c37da1f164abad83aff5d9da176c3a859badfb" exitCode=60 Oct 30 21:45:21.644398 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.644073 5801 generic.go:356] "Generic (PLEG): container finished" podID="e76f09d3-4c42-4917-92cb-0f61d7cb381e" containerID="5344723816eada7ce2cf6d29bd4ef53a5b9a1c1fdd29c96b72b71b6193f58b31" exitCode=60 Oct 30 21:45:21.646997 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.645945 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8" Oct 30 21:45:21.651249 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.651221 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6" event={"ID":"245e03b0-e13d-42fd-a32f-33891baa7f44","Type":"ContainerDied","Data":"058aeada96f7aba93c3c83d4a4c37da1f164abad83aff5d9da176c3a859badfb"} Oct 30 21:45:21.651445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.651430 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:45:21.652035 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.652013 5801 scope.go:122] "RemoveContainer" containerID="058aeada96f7aba93c3c83d4a4c37da1f164abad83aff5d9da176c3a859badfb" Oct 30 21:45:21.652608 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:21.652582 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6_odh-ai-gateway-infra(245e03b0-e13d-42fd-a32f-33891baa7f44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6" podUID="245e03b0-e13d-42fd-a32f-33891baa7f44" Oct 30 21:45:21.656883 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.651578 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb" event={"ID":"e76f09d3-4c42-4917-92cb-0f61d7cb381e","Type":"ContainerDied","Data":"5344723816eada7ce2cf6d29bd4ef53a5b9a1c1fdd29c96b72b71b6193f58b31"} Oct 30 21:45:21.657277 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.657259 5801 scope.go:122] "RemoveContainer" containerID="5344723816eada7ce2cf6d29bd4ef53a5b9a1c1fdd29c96b72b71b6193f58b31" Oct 30 21:45:21.657605 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:21.657582 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb_odh-ai-gateway-infra(e76f09d3-4c42-4917-92cb-0f61d7cb381e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb" podUID="e76f09d3-4c42-4917-92cb-0f61d7cb381e" Oct 30 21:45:21.661473 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.661442 5801 generic.go:356] "Generic (PLEG): container finished" podID="51fc9f9d-9a00-4001-b258-7f7850297a0e" containerID="abff2f7f94962d11299170a694baa7e43f10b08f012e624c8d1908bc8194ba7d" exitCode=60 Oct 30 21:45:21.661560 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.661511 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9" event={"ID":"51fc9f9d-9a00-4001-b258-7f7850297a0e","Type":"ContainerDied","Data":"abff2f7f94962d11299170a694baa7e43f10b08f012e624c8d1908bc8194ba7d"} Oct 30 21:45:21.662154 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.662134 5801 scope.go:122] "RemoveContainer" containerID="abff2f7f94962d11299170a694baa7e43f10b08f012e624c8d1908bc8194ba7d" Oct 30 21:45:21.662692 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:21.662666 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9_odh-ai-gateway-infra(51fc9f9d-9a00-4001-b258-7f7850297a0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9" podUID="51fc9f9d-9a00-4001-b258-7f7850297a0e" Oct 30 21:45:21.669201 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.669114 5801 scope.go:122] "RemoveContainer" containerID="ad3e544fe1e02bc0475f8df9948ea4c6ab94e71ff1a846a89796f5406efa5ee8" Oct 30 21:45:21.683802 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.683624 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp" Oct 30 21:45:21.707325 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.707294 5801 scope.go:122] "RemoveContainer" containerID="c638dc71787df96e67a89b24d3e58d86f87d5d5f2b350f6f8b5b1a50c1dfd10b" Oct 30 21:45:21.732444 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.732410 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/863ab859-100b-4fe9-843c-8390ebeab018-kube-api-access-mx8sq\" (UniqueName: \"kubernetes.io/projected/863ab859-100b-4fe9-843c-8390ebeab018-kube-api-access-mx8sq\") pod \"863ab859-100b-4fe9-843c-8390ebeab018\" (UID: \"863ab859-100b-4fe9-843c-8390ebeab018\") " Oct 30 21:45:21.732579 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.732490 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/863ab859-100b-4fe9-843c-8390ebeab018-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/863ab859-100b-4fe9-843c-8390ebeab018-maas-api-service-ca\") pod \"863ab859-100b-4fe9-843c-8390ebeab018\" (UID: \"863ab859-100b-4fe9-843c-8390ebeab018\") " Oct 30 21:45:21.732652 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.732639 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58a71e2-3701-43d1-a341-58e5bef8732d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f58a71e2-3701-43d1-a341-58e5bef8732d-maas-api-service-ca\") pod \"f58a71e2-3701-43d1-a341-58e5bef8732d\" (UID: \"f58a71e2-3701-43d1-a341-58e5bef8732d\") " Oct 30 21:45:21.732789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.732748 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58a71e2-3701-43d1-a341-58e5bef8732d-kube-api-access-mmdnx\" (UniqueName: \"kubernetes.io/projected/f58a71e2-3701-43d1-a341-58e5bef8732d-kube-api-access-mmdnx\") pod \"f58a71e2-3701-43d1-a341-58e5bef8732d\" (UID: \"f58a71e2-3701-43d1-a341-58e5bef8732d\") " Oct 30 21:45:21.740373 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.737689 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/863ab859-100b-4fe9-843c-8390ebeab018-maas-api-service-ca" pod "863ab859-100b-4fe9-843c-8390ebeab018" (UID: "863ab859-100b-4fe9-843c-8390ebeab018"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:21.740373 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.739570 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58a71e2-3701-43d1-a341-58e5bef8732d-maas-api-service-ca" pod "f58a71e2-3701-43d1-a341-58e5bef8732d" (UID: "f58a71e2-3701-43d1-a341-58e5bef8732d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:21.742138 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.742060 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/863ab859-100b-4fe9-843c-8390ebeab018-maas-api-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:21.742472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.742455 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f58a71e2-3701-43d1-a341-58e5bef8732d-maas-api-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:21.753520 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.748071 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58a71e2-3701-43d1-a341-58e5bef8732d-kube-api-access-mmdnx" pod "f58a71e2-3701-43d1-a341-58e5bef8732d" (UID: "f58a71e2-3701-43d1-a341-58e5bef8732d"). InnerVolumeSpecName "kube-api-access-mmdnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:21.753520 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.749739 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863ab859-100b-4fe9-843c-8390ebeab018-kube-api-access-mx8sq" pod "863ab859-100b-4fe9-843c-8390ebeab018" (UID: "863ab859-100b-4fe9-843c-8390ebeab018"). InnerVolumeSpecName "kube-api-access-mx8sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:21.785904 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.785858 5801 scope.go:122] "RemoveContainer" containerID="4cba48dc748e474d599b1b89c28c412bb1922b5b0b92177b65917e781c0ed229" Oct 30 21:45:21.891833 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.884033 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4ad7312-8811-476b-9e22-2d062cfa5508-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ad7312-8811-476b-9e22-2d062cfa5508-maas-api-service-ca\") pod \"b4ad7312-8811-476b-9e22-2d062cfa5508\" (UID: \"b4ad7312-8811-476b-9e22-2d062cfa5508\") " Oct 30 21:45:21.891833 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.884134 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ad7312-8811-476b-9e22-2d062cfa5508-kube-api-access-5vkbb\" (UniqueName: \"kubernetes.io/projected/b4ad7312-8811-476b-9e22-2d062cfa5508-kube-api-access-5vkbb\") pod \"b4ad7312-8811-476b-9e22-2d062cfa5508\" (UID: \"b4ad7312-8811-476b-9e22-2d062cfa5508\") " Oct 30 21:45:21.891833 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.884866 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmdnx\" (UniqueName: \"kubernetes.io/projected/f58a71e2-3701-43d1-a341-58e5bef8732d-kube-api-access-mmdnx\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:21.891833 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.884889 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx8sq\" (UniqueName: \"kubernetes.io/projected/863ab859-100b-4fe9-843c-8390ebeab018-kube-api-access-mx8sq\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:21.891833 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.888509 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ad7312-8811-476b-9e22-2d062cfa5508-maas-api-service-ca" pod "b4ad7312-8811-476b-9e22-2d062cfa5508" (UID: "b4ad7312-8811-476b-9e22-2d062cfa5508"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:21.891833 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.889151 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ad7312-8811-476b-9e22-2d062cfa5508-kube-api-access-5vkbb" pod "b4ad7312-8811-476b-9e22-2d062cfa5508" (UID: "b4ad7312-8811-476b-9e22-2d062cfa5508"). InnerVolumeSpecName "kube-api-access-5vkbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:21.988035 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.987419 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ad7312-8811-476b-9e22-2d062cfa5508-maas-api-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:21.988035 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.987481 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vkbb\" (UniqueName: \"kubernetes.io/projected/b4ad7312-8811-476b-9e22-2d062cfa5508-kube-api-access-5vkbb\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:21.988645 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:21.988622 5801 scope.go:122] "RemoveContainer" containerID="b07c027b043693617bbe3861271fc7c66269fe1329acf8a303db77a997f4da18" Oct 30 21:45:22.031096 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.031052 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:22.207548 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.203728 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw" Oct 30 21:45:22.237989 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.237914 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286" Oct 30 21:45:22.294018 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.293985 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154205a2-c374-4c7e-b869-c48c8f488db8-kube-api-access-4b4wc\" (UniqueName: \"kubernetes.io/projected/154205a2-c374-4c7e-b869-c48c8f488db8-kube-api-access-4b4wc\") pod \"154205a2-c374-4c7e-b869-c48c8f488db8\" (UID: \"154205a2-c374-4c7e-b869-c48c8f488db8\") " Oct 30 21:45:22.294178 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.294060 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/154205a2-c374-4c7e-b869-c48c8f488db8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/154205a2-c374-4c7e-b869-c48c8f488db8-maas-api-service-ca\") pod \"154205a2-c374-4c7e-b869-c48c8f488db8\" (UID: \"154205a2-c374-4c7e-b869-c48c8f488db8\") " Oct 30 21:45:22.294296 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.294280 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15e03e4b-deae-4513-83d2-efd7f5538d24-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15e03e4b-deae-4513-83d2-efd7f5538d24-maas-api-service-ca\") pod \"15e03e4b-deae-4513-83d2-efd7f5538d24\" (UID: \"15e03e4b-deae-4513-83d2-efd7f5538d24\") " Oct 30 21:45:22.294355 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.294339 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15e03e4b-deae-4513-83d2-efd7f5538d24-kube-api-access-5dbgs\" (UniqueName: \"kubernetes.io/projected/15e03e4b-deae-4513-83d2-efd7f5538d24-kube-api-access-5dbgs\") pod \"15e03e4b-deae-4513-83d2-efd7f5538d24\" (UID: \"15e03e4b-deae-4513-83d2-efd7f5538d24\") " Oct 30 21:45:22.295239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.295208 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15e03e4b-deae-4513-83d2-efd7f5538d24-maas-api-service-ca" pod "15e03e4b-deae-4513-83d2-efd7f5538d24" (UID: "15e03e4b-deae-4513-83d2-efd7f5538d24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:22.295398 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.295377 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/154205a2-c374-4c7e-b869-c48c8f488db8-maas-api-service-ca" pod "154205a2-c374-4c7e-b869-c48c8f488db8" (UID: "154205a2-c374-4c7e-b869-c48c8f488db8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:22.297531 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.297508 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e03e4b-deae-4513-83d2-efd7f5538d24-kube-api-access-5dbgs" pod "15e03e4b-deae-4513-83d2-efd7f5538d24" (UID: "15e03e4b-deae-4513-83d2-efd7f5538d24"). InnerVolumeSpecName "kube-api-access-5dbgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:22.297531 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.297538 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154205a2-c374-4c7e-b869-c48c8f488db8-kube-api-access-4b4wc" pod "154205a2-c374-4c7e-b869-c48c8f488db8" (UID: "154205a2-c374-4c7e-b869-c48c8f488db8"). InnerVolumeSpecName "kube-api-access-4b4wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:22.397458 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.397366 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15e03e4b-deae-4513-83d2-efd7f5538d24-maas-api-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:22.397458 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.397403 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dbgs\" (UniqueName: \"kubernetes.io/projected/15e03e4b-deae-4513-83d2-efd7f5538d24-kube-api-access-5dbgs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:22.397458 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.397418 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b4wc\" (UniqueName: \"kubernetes.io/projected/154205a2-c374-4c7e-b869-c48c8f488db8-kube-api-access-4b4wc\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:22.397458 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.397432 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/154205a2-c374-4c7e-b869-c48c8f488db8-maas-api-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:22.678679 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.678644 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52" Oct 30 21:45:22.679224 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.678660 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52" event={"ID":"f58a71e2-3701-43d1-a341-58e5bef8732d","Type":"ContainerDied","Data":"16384554279a17c69af2cf33856d42daf60bac619a8ba1d645926cf5f99155e5"} Oct 30 21:45:22.679400 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.679378 5801 scope.go:122] "RemoveContainer" containerID="511257441a4b0b450847060919134b941dd604b62765dadbd21df6a2b368631c" Oct 30 21:45:22.682683 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.682618 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286" Oct 30 21:45:22.682683 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.682632 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286" event={"ID":"154205a2-c374-4c7e-b869-c48c8f488db8","Type":"ContainerDied","Data":"55ec978bbbe0cb7873a451df967f82e7f2678b9f6b8b48f3469dc2d03ee44810"} Oct 30 21:45:22.697586 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.697353 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw" Oct 30 21:45:22.697586 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.697355 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw" event={"ID":"15e03e4b-deae-4513-83d2-efd7f5538d24","Type":"ContainerDied","Data":"77a429e5a824d955476c64eaa820fb857c96ac9cbe0621224ed98e7923e8220f"} Oct 30 21:45:22.701689 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.700939 5801 scope.go:122] "RemoveContainer" containerID="c6b1e9094fa85c41f02da649311cc0afdddfbd432bc0bab89a54c1115cb857f0" Oct 30 21:45:22.707086 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.707037 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8" event={"ID":"863ab859-100b-4fe9-843c-8390ebeab018","Type":"ContainerDied","Data":"f3164346be13d5ee86bbe9fccff1e7d0c8fc1265ad94a9da982a0d516d4d50f2"} Oct 30 21:45:22.708403 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.707180 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8" Oct 30 21:45:22.719201 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.719169 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp" event={"ID":"b4ad7312-8811-476b-9e22-2d062cfa5508","Type":"ContainerDied","Data":"9be5057949c7eeb98c90f3eed93c332ee9c8fd94ef731a5778c68c40bec8957c"} Oct 30 21:45:22.719336 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.719317 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp" Oct 30 21:45:22.729089 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.729065 5801 scope.go:122] "RemoveContainer" containerID="5cec1a5ecb450a8e1d12a75d45b948bba1c5244f8a108618ac2def22359e8dc7" Oct 30 21:45:22.731479 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.731457 5801 scope.go:122] "RemoveContainer" containerID="3b0fe246887db49a3fba184e07b776fb8ba8bb6176fa12cefd4b862219e8a732" Oct 30 21:45:22.732046 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:22.731901 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8_odh-ai-gateway-infra(e4314f13-77fd-48d6-9387-3648850025da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8" podUID="e4314f13-77fd-48d6-9387-3648850025da" Oct 30 21:45:22.735422 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.735385 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286"] Oct 30 21:45:22.744895 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.744872 5801 scope.go:122] "RemoveContainer" containerID="3d11c823d2d984142c51a58b9f63f8d2173e753d92d877e8d0d5d51474076801" Oct 30 21:45:22.751916 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.751894 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2b36c1-29889945-zb286"] Oct 30 21:45:22.755997 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.755981 5801 scope.go:122] "RemoveContainer" containerID="8acc4ff11450ad2671ab19c7bebfbf7392ac2d8a7bbd3e2e34653ee91f173d29" Oct 30 21:45:22.758403 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.758383 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52"] Oct 30 21:45:22.765144 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.765120 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1a81ae-29889945-wvz52"] Oct 30 21:45:22.781417 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.781388 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp"] Oct 30 21:45:22.786450 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.786431 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-caa3dc-29889945-crtqp"] Oct 30 21:45:22.797241 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.797223 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8"] Oct 30 21:45:22.805104 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.805085 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0fa864-29889945-l5xd8"] Oct 30 21:45:22.819945 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.819925 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw"] Oct 30 21:45:22.824860 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:22.824837 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1f725d-29889945-zw2lw"] Oct 30 21:45:23.203720 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.203686 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:23.271606 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.271571 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8"] Oct 30 21:45:23.359860 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.359823 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb"] Oct 30 21:45:23.476901 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.476824 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9"] Oct 30 21:45:23.524321 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.521764 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6"] Oct 30 21:45:23.549412 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.549385 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="154205a2-c374-4c7e-b869-c48c8f488db8" path="/var/lib/kubelet/pods/154205a2-c374-4c7e-b869-c48c8f488db8/volumes" Oct 30 21:45:23.556653 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.556606 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15e03e4b-deae-4513-83d2-efd7f5538d24" path="/var/lib/kubelet/pods/15e03e4b-deae-4513-83d2-efd7f5538d24/volumes" Oct 30 21:45:23.557820 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.557793 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="863ab859-100b-4fe9-843c-8390ebeab018" path="/var/lib/kubelet/pods/863ab859-100b-4fe9-843c-8390ebeab018/volumes" Oct 30 21:45:23.560370 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.560342 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4ad7312-8811-476b-9e22-2d062cfa5508" path="/var/lib/kubelet/pods/b4ad7312-8811-476b-9e22-2d062cfa5508/volumes" Oct 30 21:45:23.564815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.561034 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58a71e2-3701-43d1-a341-58e5bef8732d" path="/var/lib/kubelet/pods/f58a71e2-3701-43d1-a341-58e5bef8732d/volumes" Oct 30 21:45:23.602802 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.600618 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb" Oct 30 21:45:23.771191 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.771143 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e76f09d3-4c42-4917-92cb-0f61d7cb381e-kube-api-access-x5nzz\" (UniqueName: \"kubernetes.io/projected/e76f09d3-4c42-4917-92cb-0f61d7cb381e-kube-api-access-x5nzz\") pod \"e76f09d3-4c42-4917-92cb-0f61d7cb381e\" (UID: \"e76f09d3-4c42-4917-92cb-0f61d7cb381e\") " Oct 30 21:45:23.772848 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.772822 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e76f09d3-4c42-4917-92cb-0f61d7cb381e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e76f09d3-4c42-4917-92cb-0f61d7cb381e-maas-api-service-ca\") pod \"e76f09d3-4c42-4917-92cb-0f61d7cb381e\" (UID: \"e76f09d3-4c42-4917-92cb-0f61d7cb381e\") " Oct 30 21:45:23.774320 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.774279 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e76f09d3-4c42-4917-92cb-0f61d7cb381e-maas-api-service-ca" pod "e76f09d3-4c42-4917-92cb-0f61d7cb381e" (UID: "e76f09d3-4c42-4917-92cb-0f61d7cb381e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:23.792252 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.792213 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76f09d3-4c42-4917-92cb-0f61d7cb381e-kube-api-access-x5nzz" pod "e76f09d3-4c42-4917-92cb-0f61d7cb381e" (UID: "e76f09d3-4c42-4917-92cb-0f61d7cb381e"). InnerVolumeSpecName "kube-api-access-x5nzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:23.795561 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.795539 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb" Oct 30 21:45:23.795802 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.795538 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb" event={"ID":"e76f09d3-4c42-4917-92cb-0f61d7cb381e","Type":"ContainerDied","Data":"676eaecc7e14b9d5844bffecee6c04608c05875b8a5f293ae7fb5b3c9edd819d"} Oct 30 21:45:23.795883 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.795873 5801 scope.go:122] "RemoveContainer" containerID="5344723816eada7ce2cf6d29bd4ef53a5b9a1c1fdd29c96b72b71b6193f58b31" Oct 30 21:45:23.864027 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.863930 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9" Oct 30 21:45:23.870636 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.870606 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6" Oct 30 21:45:23.875462 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.875433 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e76f09d3-4c42-4917-92cb-0f61d7cb381e-maas-api-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:23.875462 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.875459 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5nzz\" (UniqueName: \"kubernetes.io/projected/e76f09d3-4c42-4917-92cb-0f61d7cb381e-kube-api-access-x5nzz\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:23.876385 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.876363 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb"] Oct 30 21:45:23.888334 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.888309 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-db0b54-29889945-bntmb"] Oct 30 21:45:23.979172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.979137 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51fc9f9d-9a00-4001-b258-7f7850297a0e-kube-api-access-2xjp4\" (UniqueName: \"kubernetes.io/projected/51fc9f9d-9a00-4001-b258-7f7850297a0e-kube-api-access-2xjp4\") pod \"51fc9f9d-9a00-4001-b258-7f7850297a0e\" (UID: \"51fc9f9d-9a00-4001-b258-7f7850297a0e\") " Oct 30 21:45:23.979343 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.979216 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/245e03b0-e13d-42fd-a32f-33891baa7f44-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/245e03b0-e13d-42fd-a32f-33891baa7f44-maas-api-service-ca\") pod \"245e03b0-e13d-42fd-a32f-33891baa7f44\" (UID: \"245e03b0-e13d-42fd-a32f-33891baa7f44\") " Oct 30 21:45:23.979343 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.979274 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51fc9f9d-9a00-4001-b258-7f7850297a0e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51fc9f9d-9a00-4001-b258-7f7850297a0e-maas-api-service-ca\") pod \"51fc9f9d-9a00-4001-b258-7f7850297a0e\" (UID: \"51fc9f9d-9a00-4001-b258-7f7850297a0e\") " Oct 30 21:45:23.979493 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.979365 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/245e03b0-e13d-42fd-a32f-33891baa7f44-kube-api-access-kw8sm\" (UniqueName: \"kubernetes.io/projected/245e03b0-e13d-42fd-a32f-33891baa7f44-kube-api-access-kw8sm\") pod \"245e03b0-e13d-42fd-a32f-33891baa7f44\" (UID: \"245e03b0-e13d-42fd-a32f-33891baa7f44\") " Oct 30 21:45:23.981438 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.981408 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/245e03b0-e13d-42fd-a32f-33891baa7f44-maas-api-service-ca" pod "245e03b0-e13d-42fd-a32f-33891baa7f44" (UID: "245e03b0-e13d-42fd-a32f-33891baa7f44"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:23.982114 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.982087 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51fc9f9d-9a00-4001-b258-7f7850297a0e-maas-api-service-ca" pod "51fc9f9d-9a00-4001-b258-7f7850297a0e" (UID: "51fc9f9d-9a00-4001-b258-7f7850297a0e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:23.998333 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.998283 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245e03b0-e13d-42fd-a32f-33891baa7f44-kube-api-access-kw8sm" pod "245e03b0-e13d-42fd-a32f-33891baa7f44" (UID: "245e03b0-e13d-42fd-a32f-33891baa7f44"). InnerVolumeSpecName "kube-api-access-kw8sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:23.998596 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:23.998565 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fc9f9d-9a00-4001-b258-7f7850297a0e-kube-api-access-2xjp4" pod "51fc9f9d-9a00-4001-b258-7f7850297a0e" (UID: "51fc9f9d-9a00-4001-b258-7f7850297a0e"). InnerVolumeSpecName "kube-api-access-2xjp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:24.028896 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.028840 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8" Oct 30 21:45:24.046233 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.046198 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:24.086391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.086357 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4314f13-77fd-48d6-9387-3648850025da-kube-api-access-nlwb8\" (UniqueName: \"kubernetes.io/projected/e4314f13-77fd-48d6-9387-3648850025da-kube-api-access-nlwb8\") pod \"e4314f13-77fd-48d6-9387-3648850025da\" (UID: \"e4314f13-77fd-48d6-9387-3648850025da\") " Oct 30 21:45:24.086559 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.086443 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4314f13-77fd-48d6-9387-3648850025da-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4314f13-77fd-48d6-9387-3648850025da-maas-api-service-ca\") pod \"e4314f13-77fd-48d6-9387-3648850025da\" (UID: \"e4314f13-77fd-48d6-9387-3648850025da\") " Oct 30 21:45:24.087229 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.087204 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xjp4\" (UniqueName: \"kubernetes.io/projected/51fc9f9d-9a00-4001-b258-7f7850297a0e-kube-api-access-2xjp4\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:24.087229 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.087230 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/245e03b0-e13d-42fd-a32f-33891baa7f44-maas-api-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:24.087430 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.087243 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51fc9f9d-9a00-4001-b258-7f7850297a0e-maas-api-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:24.087430 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.087256 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw8sm\" (UniqueName: \"kubernetes.io/projected/245e03b0-e13d-42fd-a32f-33891baa7f44-kube-api-access-kw8sm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:24.087612 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.087592 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4314f13-77fd-48d6-9387-3648850025da-maas-api-service-ca" pod "e4314f13-77fd-48d6-9387-3648850025da" (UID: "e4314f13-77fd-48d6-9387-3648850025da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:24.090267 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.090242 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4314f13-77fd-48d6-9387-3648850025da-kube-api-access-nlwb8" pod "e4314f13-77fd-48d6-9387-3648850025da" (UID: "e4314f13-77fd-48d6-9387-3648850025da"). InnerVolumeSpecName "kube-api-access-nlwb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:24.188791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.188704 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlwb8\" (UniqueName: \"kubernetes.io/projected/e4314f13-77fd-48d6-9387-3648850025da-kube-api-access-nlwb8\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:24.188791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.188738 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4314f13-77fd-48d6-9387-3648850025da-maas-api-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:24.815788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.815707 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8" Oct 30 21:45:24.815788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.815710 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8" event={"ID":"e4314f13-77fd-48d6-9387-3648850025da","Type":"ContainerDied","Data":"bb56dd736b58786cc378c91df2b6fbeedb737dab1949f0a0fcbcd4b280919071"} Oct 30 21:45:24.816283 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.815788 5801 scope.go:122] "RemoveContainer" containerID="3b0fe246887db49a3fba184e07b776fb8ba8bb6176fa12cefd4b862219e8a732" Oct 30 21:45:24.817513 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.817486 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6" Oct 30 21:45:24.817615 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.817541 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6" event={"ID":"245e03b0-e13d-42fd-a32f-33891baa7f44","Type":"ContainerDied","Data":"46ef4d6bc752862fb164c39edbd07261b1037efc8e8a73c8ac1972edf79f9750"} Oct 30 21:45:24.821142 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.821114 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9" Oct 30 21:45:24.821248 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.821149 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9" event={"ID":"51fc9f9d-9a00-4001-b258-7f7850297a0e","Type":"ContainerDied","Data":"5f07b8d9b0ac021b5221a19de9e1554d464503115f00fe8827c98f45daf3b169"} Oct 30 21:45:24.831492 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.831129 5801 scope.go:122] "RemoveContainer" containerID="058aeada96f7aba93c3c83d4a4c37da1f164abad83aff5d9da176c3a859badfb" Oct 30 21:45:24.842422 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.842404 5801 scope.go:122] "RemoveContainer" containerID="abff2f7f94962d11299170a694baa7e43f10b08f012e624c8d1908bc8194ba7d" Oct 30 21:45:24.846870 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.846841 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8"] Oct 30 21:45:24.852063 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.852034 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:24.853449 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.853426 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d8034-29889945-hbff8"] Oct 30 21:45:24.863058 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.863032 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9"] Oct 30 21:45:24.868107 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.868088 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-000f64-29889945-qmzv9"] Oct 30 21:45:24.883014 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.882993 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6"] Oct 30 21:45:24.892397 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:24.891298 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-0b412a-29889945-tf8t6"] Oct 30 21:45:25.528396 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:25.528361 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="245e03b0-e13d-42fd-a32f-33891baa7f44" path="/var/lib/kubelet/pods/245e03b0-e13d-42fd-a32f-33891baa7f44/volumes" Oct 30 21:45:25.528818 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:25.528803 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51fc9f9d-9a00-4001-b258-7f7850297a0e" path="/var/lib/kubelet/pods/51fc9f9d-9a00-4001-b258-7f7850297a0e/volumes" Oct 30 21:45:25.529204 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:25.529191 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4314f13-77fd-48d6-9387-3648850025da" path="/var/lib/kubelet/pods/e4314f13-77fd-48d6-9387-3648850025da/volumes" Oct 30 21:45:25.529861 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:25.529843 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e76f09d3-4c42-4917-92cb-0f61d7cb381e" path="/var/lib/kubelet/pods/e76f09d3-4c42-4917-92cb-0f61d7cb381e/volumes" Oct 30 21:45:25.760156 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:25.760123 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:25.866226 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:25.866190 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:26.034269 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:26.034233 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-548bdc16-t2286"] Oct 30 21:45:26.041552 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:26.041516 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-548bdc16-t2286"] Oct 30 21:45:26.684802 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:26.684474 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:26.854175 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:26.854142 5801 generic.go:356] "Generic (PLEG): container finished" podID="2fe17509-a317-4eae-8edd-de1383398615" containerID="990d404bc28815dc548b2efe8ed1b456d8f7196f81375af3e75dd80ac7e15524" exitCode=0 Oct 30 21:45:26.854415 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:26.854219 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" event={"ID":"2fe17509-a317-4eae-8edd-de1383398615","Type":"ContainerDied","Data":"990d404bc28815dc548b2efe8ed1b456d8f7196f81375af3e75dd80ac7e15524"} Oct 30 21:45:26.854730 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:26.854712 5801 scope.go:122] "RemoveContainer" containerID="990d404bc28815dc548b2efe8ed1b456d8f7196f81375af3e75dd80ac7e15524" Oct 30 21:45:26.856422 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:26.856392 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:26.856533 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:26.856476 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:26.856533 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:26.856527 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:27.435968 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:27.435936 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:27.527551 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:27.527519 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45db6891-8460-4a7f-a9ef-ed41e3ce06c1" path="/var/lib/kubelet/pods/45db6891-8460-4a7f-a9ef-ed41e3ce06c1/volumes" Oct 30 21:45:27.780285 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:27.780180 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:27.885226 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:27.885184 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" event={"ID":"2fe17509-a317-4eae-8edd-de1383398615","Type":"ContainerStarted","Data":"c9fc4971acbfbd843a554bf516ba427b62331da91968c3ba02d505b5705d3f1b"} Oct 30 21:45:27.888407 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:27.888373 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:27.989669 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:27.989637 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:28.280898 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:28.280862 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:28.830384 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:28.830341 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:45:28.831841 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:28.831806 5801 patch_prober.go:29] interesting pod/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 21:45:28.831968 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:28.831879 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" podUID="2fe17509-a317-4eae-8edd-de1383398615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 21:45:29.047885 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:29.047849 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2828ee03-n9ncr"] Oct 30 21:45:29.050287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:29.050254 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:29.079195 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:29.079160 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2828ee03-n9ncr"] Oct 30 21:45:29.241870 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:29.241700 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:29.357776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:29.347013 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:29.481779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:29.481643 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:45:29.484061 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:29.482164 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:29.541652 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:29.541576 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34a80fd8-1d28-42c0-ba24-7a3fb3db1886" path="/var/lib/kubelet/pods/34a80fd8-1d28-42c0-ba24-7a3fb3db1886/volumes" Oct 30 21:45:29.830029 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:29.830000 5801 patch_prober.go:29] interesting pod/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 21:45:29.830199 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:29.830059 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" podUID="2fe17509-a317-4eae-8edd-de1383398615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 21:45:30.308460 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:45:30.308426 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:45:30.324210 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.324180 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:30.830021 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.829989 5801 patch_prober.go:29] interesting pod/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 21:45:30.830221 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.830049 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" podUID="2fe17509-a317-4eae-8edd-de1383398615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 21:45:30.885285 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.884878 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b1b1870-xwj5b"] Oct 30 21:45:30.885986 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.885954 5801 cpu_manager.go:414] "containerMap: removing container" podUID="245e03b0-e13d-42fd-a32f-33891baa7f44" containerName="cleanup" Oct 30 21:45:30.885986 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.885977 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="245e03b0-e13d-42fd-a32f-33891baa7f44" containerName="cleanup" Oct 30 21:45:30.885986 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.885992 5801 cpu_manager.go:414] "containerMap: removing container" podUID="15e03e4b-deae-4513-83d2-efd7f5538d24" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.885999 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15e03e4b-deae-4513-83d2-efd7f5538d24" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886017 5801 cpu_manager.go:414] "containerMap: removing container" podUID="e4314f13-77fd-48d6-9387-3648850025da" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886027 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4314f13-77fd-48d6-9387-3648850025da" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886048 5801 cpu_manager.go:414] "containerMap: removing container" podUID="f58a71e2-3701-43d1-a341-58e5bef8732d" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886055 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58a71e2-3701-43d1-a341-58e5bef8732d" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886065 5801 cpu_manager.go:414] "containerMap: removing container" podUID="15e03e4b-deae-4513-83d2-efd7f5538d24" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886072 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15e03e4b-deae-4513-83d2-efd7f5538d24" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886091 5801 cpu_manager.go:414] "containerMap: removing container" podUID="245e03b0-e13d-42fd-a32f-33891baa7f44" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886097 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="245e03b0-e13d-42fd-a32f-33891baa7f44" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886111 5801 cpu_manager.go:414] "containerMap: removing container" podUID="b4ad7312-8811-476b-9e22-2d062cfa5508" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886118 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ad7312-8811-476b-9e22-2d062cfa5508" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886129 5801 cpu_manager.go:414] "containerMap: removing container" podUID="f58a71e2-3701-43d1-a341-58e5bef8732d" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886136 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58a71e2-3701-43d1-a341-58e5bef8732d" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886147 5801 cpu_manager.go:414] "containerMap: removing container" podUID="863ab859-100b-4fe9-843c-8390ebeab018" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886153 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="863ab859-100b-4fe9-843c-8390ebeab018" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886170 5801 cpu_manager.go:414] "containerMap: removing container" podUID="863ab859-100b-4fe9-843c-8390ebeab018" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886176 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="863ab859-100b-4fe9-843c-8390ebeab018" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886187 5801 cpu_manager.go:414] "containerMap: removing container" podUID="e76f09d3-4c42-4917-92cb-0f61d7cb381e" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886194 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e76f09d3-4c42-4917-92cb-0f61d7cb381e" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886204 5801 cpu_manager.go:414] "containerMap: removing container" podUID="154205a2-c374-4c7e-b869-c48c8f488db8" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886210 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154205a2-c374-4c7e-b869-c48c8f488db8" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886219 5801 cpu_manager.go:414] "containerMap: removing container" podUID="e4314f13-77fd-48d6-9387-3648850025da" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886228 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4314f13-77fd-48d6-9387-3648850025da" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886238 5801 cpu_manager.go:414] "containerMap: removing container" podUID="154205a2-c374-4c7e-b869-c48c8f488db8" containerName="cleanup" Oct 30 21:45:30.886239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886244 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154205a2-c374-4c7e-b869-c48c8f488db8" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886256 5801 cpu_manager.go:414] "containerMap: removing container" podUID="863ab859-100b-4fe9-843c-8390ebeab018" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886263 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="863ab859-100b-4fe9-843c-8390ebeab018" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886275 5801 cpu_manager.go:414] "containerMap: removing container" podUID="51fc9f9d-9a00-4001-b258-7f7850297a0e" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886281 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51fc9f9d-9a00-4001-b258-7f7850297a0e" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886290 5801 cpu_manager.go:414] "containerMap: removing container" podUID="e76f09d3-4c42-4917-92cb-0f61d7cb381e" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886296 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e76f09d3-4c42-4917-92cb-0f61d7cb381e" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886306 5801 cpu_manager.go:414] "containerMap: removing container" podUID="b4ad7312-8811-476b-9e22-2d062cfa5508" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886312 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ad7312-8811-476b-9e22-2d062cfa5508" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886326 5801 cpu_manager.go:414] "containerMap: removing container" podUID="154205a2-c374-4c7e-b869-c48c8f488db8" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886333 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154205a2-c374-4c7e-b869-c48c8f488db8" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886341 5801 cpu_manager.go:414] "containerMap: removing container" podUID="51fc9f9d-9a00-4001-b258-7f7850297a0e" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886347 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51fc9f9d-9a00-4001-b258-7f7850297a0e" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886365 5801 cpu_manager.go:414] "containerMap: removing container" podUID="b4ad7312-8811-476b-9e22-2d062cfa5508" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886372 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ad7312-8811-476b-9e22-2d062cfa5508" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886384 5801 cpu_manager.go:414] "containerMap: removing container" podUID="e76f09d3-4c42-4917-92cb-0f61d7cb381e" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886390 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e76f09d3-4c42-4917-92cb-0f61d7cb381e" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886545 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="245e03b0-e13d-42fd-a32f-33891baa7f44" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886555 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="15e03e4b-deae-4513-83d2-efd7f5538d24" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886564 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4314f13-77fd-48d6-9387-3648850025da" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886575 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58a71e2-3701-43d1-a341-58e5bef8732d" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886584 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="245e03b0-e13d-42fd-a32f-33891baa7f44" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886592 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="245e03b0-e13d-42fd-a32f-33891baa7f44" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886605 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ad7312-8811-476b-9e22-2d062cfa5508" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886614 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58a71e2-3701-43d1-a341-58e5bef8732d" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886622 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="863ab859-100b-4fe9-843c-8390ebeab018" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886629 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="863ab859-100b-4fe9-843c-8390ebeab018" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886636 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="e76f09d3-4c42-4917-92cb-0f61d7cb381e" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886645 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="154205a2-c374-4c7e-b869-c48c8f488db8" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886653 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4314f13-77fd-48d6-9387-3648850025da" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886661 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="154205a2-c374-4c7e-b869-c48c8f488db8" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886669 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="51fc9f9d-9a00-4001-b258-7f7850297a0e" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886677 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="e76f09d3-4c42-4917-92cb-0f61d7cb381e" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886686 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ad7312-8811-476b-9e22-2d062cfa5508" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886695 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="154205a2-c374-4c7e-b869-c48c8f488db8" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886704 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="51fc9f9d-9a00-4001-b258-7f7850297a0e" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886714 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ad7312-8811-476b-9e22-2d062cfa5508" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886722 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="15e03e4b-deae-4513-83d2-efd7f5538d24" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886730 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="e76f09d3-4c42-4917-92cb-0f61d7cb381e" containerName="cleanup" Oct 30 21:45:30.890266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.886739 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="51fc9f9d-9a00-4001-b258-7f7850297a0e" containerName="cleanup" Oct 30 21:45:30.901766 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.900108 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b1b1870-xwj5b"] Oct 30 21:45:30.901766 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.900220 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1b1870-xwj5b" Oct 30 21:45:30.914770 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.913068 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:45:30.920049 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.915390 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:45:30.920049 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.915950 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:45:30.920049 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.916048 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:45:30.943127 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.943102 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2720213-8297-483a-8372-880429065ae5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b1b1870-xwj5b\" (UID: \"b2720213-8297-483a-8372-880429065ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1b1870-xwj5b" Oct 30 21:45:30.943250 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.943144 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tklnj\" (UniqueName: \"kubernetes.io/projected/b2720213-8297-483a-8372-880429065ae5-kube-api-access-tklnj\") pod \"maas-api-revoke-sub-3b1b1870-xwj5b\" (UID: \"b2720213-8297-483a-8372-880429065ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1b1870-xwj5b" Oct 30 21:45:30.943352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:30.943330 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2720213-8297-483a-8372-880429065ae5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b1b1870-xwj5b\" (UID: \"b2720213-8297-483a-8372-880429065ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1b1870-xwj5b" Oct 30 21:45:31.045477 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:31.045441 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2720213-8297-483a-8372-880429065ae5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b1b1870-xwj5b\" (UID: \"b2720213-8297-483a-8372-880429065ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1b1870-xwj5b" Oct 30 21:45:31.045643 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:31.045489 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tklnj\" (UniqueName: \"kubernetes.io/projected/b2720213-8297-483a-8372-880429065ae5-kube-api-access-tklnj\") pod \"maas-api-revoke-sub-3b1b1870-xwj5b\" (UID: \"b2720213-8297-483a-8372-880429065ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1b1870-xwj5b" Oct 30 21:45:31.045643 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:31.045608 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2720213-8297-483a-8372-880429065ae5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b1b1870-xwj5b\" (UID: \"b2720213-8297-483a-8372-880429065ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1b1870-xwj5b" Oct 30 21:45:31.046091 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:31.046069 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2720213-8297-483a-8372-880429065ae5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b1b1870-xwj5b\" (UID: \"b2720213-8297-483a-8372-880429065ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1b1870-xwj5b" Oct 30 21:45:31.047979 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:31.047956 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2720213-8297-483a-8372-880429065ae5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b1b1870-xwj5b\" (UID: \"b2720213-8297-483a-8372-880429065ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1b1870-xwj5b" Oct 30 21:45:31.078388 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:31.076214 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:31.078511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:31.078434 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tklnj\" (UniqueName: \"kubernetes.io/projected/b2720213-8297-483a-8372-880429065ae5-kube-api-access-tklnj\") pod \"maas-api-revoke-sub-3b1b1870-xwj5b\" (UID: \"b2720213-8297-483a-8372-880429065ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1b1870-xwj5b" Oct 30 21:45:31.235567 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:31.235469 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1b1870-xwj5b" Oct 30 21:45:31.583346 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:31.583317 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b1b1870-xwj5b"] Oct 30 21:45:31.585262 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:45:31.585233 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2720213_8297_483a_8372_880429065ae5.slice/crio-2fa7dad8ed68f6d5ee8c8c6daae85ab4ae53e6d395d940b97f7d21332bc84a44 WatchSource:0}: Error finding container 2fa7dad8ed68f6d5ee8c8c6daae85ab4ae53e6d395d940b97f7d21332bc84a44: Status 404 returned error can't find the container with id 2fa7dad8ed68f6d5ee8c8c6daae85ab4ae53e6d395d940b97f7d21332bc84a44 Oct 30 21:45:31.830226 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:31.830195 5801 patch_prober.go:29] interesting pod/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 21:45:31.830377 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:31.830248 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" podUID="2fe17509-a317-4eae-8edd-de1383398615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 21:45:31.931145 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:31.931066 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:31.943012 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:31.942986 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1b1870-xwj5b" event={"ID":"b2720213-8297-483a-8372-880429065ae5","Type":"ContainerStarted","Data":"2fa7dad8ed68f6d5ee8c8c6daae85ab4ae53e6d395d940b97f7d21332bc84a44"} Oct 30 21:45:32.773055 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:32.772955 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:32.834776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:32.834195 5801 patch_prober.go:29] interesting pod/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 21:45:32.834776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:32.834258 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" podUID="2fe17509-a317-4eae-8edd-de1383398615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 21:45:32.885933 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:32.885853 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:32.990790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:32.990688 5801 generic.go:356] "Generic (PLEG): container finished" podID="b2720213-8297-483a-8372-880429065ae5" containerID="608ce07e38703cb3970537f001c8a68995e9500e91a9a0162ebe998874cfabc1" exitCode=0 Oct 30 21:45:32.990790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:32.990764 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1b1870-xwj5b" event={"ID":"b2720213-8297-483a-8372-880429065ae5","Type":"ContainerDied","Data":"608ce07e38703cb3970537f001c8a68995e9500e91a9a0162ebe998874cfabc1"} Oct 30 21:45:33.547501 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:33.547460 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:33.833921 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:33.833894 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:45:33.834964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:33.834944 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:45:34.209211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.209177 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1b1870-xwj5b" Oct 30 21:45:34.273054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.273016 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:34.348613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.348581 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2720213-8297-483a-8372-880429065ae5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2720213-8297-483a-8372-880429065ae5-maas-api-xks-ca\") pod \"b2720213-8297-483a-8372-880429065ae5\" (UID: \"b2720213-8297-483a-8372-880429065ae5\") " Oct 30 21:45:34.348926 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.348736 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2720213-8297-483a-8372-880429065ae5-kube-api-access-tklnj\" (UniqueName: \"kubernetes.io/projected/b2720213-8297-483a-8372-880429065ae5-kube-api-access-tklnj\") pod \"b2720213-8297-483a-8372-880429065ae5\" (UID: \"b2720213-8297-483a-8372-880429065ae5\") " Oct 30 21:45:34.348926 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.348846 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2720213-8297-483a-8372-880429065ae5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2720213-8297-483a-8372-880429065ae5-maas-api-openshift-ca\") pod \"b2720213-8297-483a-8372-880429065ae5\" (UID: \"b2720213-8297-483a-8372-880429065ae5\") " Oct 30 21:45:34.349309 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.349282 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2720213-8297-483a-8372-880429065ae5-maas-api-openshift-ca" pod "b2720213-8297-483a-8372-880429065ae5" (UID: "b2720213-8297-483a-8372-880429065ae5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:34.349719 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.349697 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2720213-8297-483a-8372-880429065ae5-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:34.351802 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.351020 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2720213-8297-483a-8372-880429065ae5-maas-api-xks-ca" pod "b2720213-8297-483a-8372-880429065ae5" (UID: "b2720213-8297-483a-8372-880429065ae5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:34.351802 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.351218 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2720213-8297-483a-8372-880429065ae5-kube-api-access-tklnj" pod "b2720213-8297-483a-8372-880429065ae5" (UID: "b2720213-8297-483a-8372-880429065ae5"). InnerVolumeSpecName "kube-api-access-tklnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:34.452242 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.452132 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2720213-8297-483a-8372-880429065ae5-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:34.452242 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.452171 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tklnj\" (UniqueName: \"kubernetes.io/projected/b2720213-8297-483a-8372-880429065ae5-kube-api-access-tklnj\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:34.954929 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.954895 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e02f8610-g9grl"] Oct 30 21:45:34.955559 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.955544 5801 cpu_manager.go:414] "containerMap: removing container" podUID="f58a71e2-3701-43d1-a341-58e5bef8732d" containerName="cleanup" Oct 30 21:45:34.955603 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.955559 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58a71e2-3701-43d1-a341-58e5bef8732d" containerName="cleanup" Oct 30 21:45:34.955603 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.955577 5801 cpu_manager.go:414] "containerMap: removing container" podUID="b2720213-8297-483a-8372-880429065ae5" containerName="revoke-keys" Oct 30 21:45:34.955603 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.955582 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2720213-8297-483a-8372-880429065ae5" containerName="revoke-keys" Oct 30 21:45:34.955603 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.955590 5801 cpu_manager.go:414] "containerMap: removing container" podUID="51fc9f9d-9a00-4001-b258-7f7850297a0e" containerName="cleanup" Oct 30 21:45:34.955603 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.955595 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51fc9f9d-9a00-4001-b258-7f7850297a0e" containerName="cleanup" Oct 30 21:45:34.955772 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.955621 5801 cpu_manager.go:414] "containerMap: removing container" podUID="15e03e4b-deae-4513-83d2-efd7f5538d24" containerName="cleanup" Oct 30 21:45:34.955772 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.955626 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15e03e4b-deae-4513-83d2-efd7f5538d24" containerName="cleanup" Oct 30 21:45:34.955772 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.955637 5801 cpu_manager.go:414] "containerMap: removing container" podUID="e4314f13-77fd-48d6-9387-3648850025da" containerName="cleanup" Oct 30 21:45:34.955772 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.955641 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4314f13-77fd-48d6-9387-3648850025da" containerName="cleanup" Oct 30 21:45:34.955772 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.955660 5801 cpu_manager.go:414] "containerMap: removing container" podUID="245e03b0-e13d-42fd-a32f-33891baa7f44" containerName="cleanup" Oct 30 21:45:34.955772 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.955665 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="245e03b0-e13d-42fd-a32f-33891baa7f44" containerName="cleanup" Oct 30 21:45:34.955960 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.955777 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58a71e2-3701-43d1-a341-58e5bef8732d" containerName="cleanup" Oct 30 21:45:34.955960 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.955785 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="863ab859-100b-4fe9-843c-8390ebeab018" containerName="cleanup" Oct 30 21:45:34.955960 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.955794 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2720213-8297-483a-8372-880429065ae5" containerName="revoke-keys" Oct 30 21:45:34.955960 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.955804 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="15e03e4b-deae-4513-83d2-efd7f5538d24" containerName="cleanup" Oct 30 21:45:34.955960 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.955810 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4314f13-77fd-48d6-9387-3648850025da" containerName="cleanup" Oct 30 21:45:34.959708 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.959690 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02f8610-g9grl" Oct 30 21:45:34.968406 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:34.968374 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e02f8610-g9grl"] Oct 30 21:45:35.024878 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:35.024849 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1b1870-xwj5b" event={"ID":"b2720213-8297-483a-8372-880429065ae5","Type":"ContainerDied","Data":"2fa7dad8ed68f6d5ee8c8c6daae85ab4ae53e6d395d940b97f7d21332bc84a44"} Oct 30 21:45:35.024878 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:35.024869 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1b1870-xwj5b" Oct 30 21:45:35.025072 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:35.024892 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fa7dad8ed68f6d5ee8c8c6daae85ab4ae53e6d395d940b97f7d21332bc84a44" Oct 30 21:45:35.040736 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:35.040713 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:35.061401 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:35.061371 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db889eca-5e32-4528-b653-c72e065bd1e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e02f8610-g9grl\" (UID: \"db889eca-5e32-4528-b653-c72e065bd1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02f8610-g9grl" Oct 30 21:45:35.061584 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:35.061565 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db889eca-5e32-4528-b653-c72e065bd1e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e02f8610-g9grl\" (UID: \"db889eca-5e32-4528-b653-c72e065bd1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02f8610-g9grl" Oct 30 21:45:35.061640 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:35.061621 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qt4d\" (UniqueName: \"kubernetes.io/projected/db889eca-5e32-4528-b653-c72e065bd1e5-kube-api-access-7qt4d\") pod \"maas-api-revoke-sub-e02f8610-g9grl\" (UID: \"db889eca-5e32-4528-b653-c72e065bd1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02f8610-g9grl" Oct 30 21:45:35.163290 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:35.163250 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db889eca-5e32-4528-b653-c72e065bd1e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e02f8610-g9grl\" (UID: \"db889eca-5e32-4528-b653-c72e065bd1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02f8610-g9grl" Oct 30 21:45:35.163480 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:35.163462 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db889eca-5e32-4528-b653-c72e065bd1e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e02f8610-g9grl\" (UID: \"db889eca-5e32-4528-b653-c72e065bd1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02f8610-g9grl" Oct 30 21:45:35.163567 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:35.163549 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qt4d\" (UniqueName: \"kubernetes.io/projected/db889eca-5e32-4528-b653-c72e065bd1e5-kube-api-access-7qt4d\") pod \"maas-api-revoke-sub-e02f8610-g9grl\" (UID: \"db889eca-5e32-4528-b653-c72e065bd1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02f8610-g9grl" Oct 30 21:45:35.164169 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:35.164146 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db889eca-5e32-4528-b653-c72e065bd1e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e02f8610-g9grl\" (UID: \"db889eca-5e32-4528-b653-c72e065bd1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02f8610-g9grl" Oct 30 21:45:35.165847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:35.165822 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db889eca-5e32-4528-b653-c72e065bd1e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e02f8610-g9grl\" (UID: \"db889eca-5e32-4528-b653-c72e065bd1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02f8610-g9grl" Oct 30 21:45:35.175805 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:35.175783 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qt4d\" (UniqueName: \"kubernetes.io/projected/db889eca-5e32-4528-b653-c72e065bd1e5-kube-api-access-7qt4d\") pod \"maas-api-revoke-sub-e02f8610-g9grl\" (UID: \"db889eca-5e32-4528-b653-c72e065bd1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02f8610-g9grl" Oct 30 21:45:35.278531 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:35.278447 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02f8610-g9grl" Oct 30 21:45:35.573855 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:45:35.573820 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb889eca_5e32_4528_b653_c72e065bd1e5.slice/crio-4c94c5775c89f4a3bbb86abf230b425c0ce25276a7a4dc3d2e4a8c71fd13ade2 WatchSource:0}: Error finding container 4c94c5775c89f4a3bbb86abf230b425c0ce25276a7a4dc3d2e4a8c71fd13ade2: Status 404 returned error can't find the container with id 4c94c5775c89f4a3bbb86abf230b425c0ce25276a7a4dc3d2e4a8c71fd13ade2 Oct 30 21:45:35.574737 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:35.574707 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e02f8610-g9grl"] Oct 30 21:45:35.730378 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:35.730348 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:36.044316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:36.044270 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12519330-wfb6k"] Oct 30 21:45:36.067239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:36.067177 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12519330-wfb6k"] Oct 30 21:45:36.067917 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:36.067876 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02f8610-g9grl" event={"ID":"db889eca-5e32-4528-b653-c72e065bd1e5","Type":"ContainerStarted","Data":"d1e0e9c07264e74f7b06ba9721ad499bc6796c1114a8fd5e28d34f3ad736d1bf"} Oct 30 21:45:36.068052 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:36.067970 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02f8610-g9grl" event={"ID":"db889eca-5e32-4528-b653-c72e065bd1e5","Type":"ContainerStarted","Data":"4c94c5775c89f4a3bbb86abf230b425c0ce25276a7a4dc3d2e4a8c71fd13ade2"} Oct 30 21:45:36.082415 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:36.082389 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:36.088099 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:36.088061 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02f8610-g9grl" podStartSLOduration=2.088049353 podStartE2EDuration="2.088049353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:36.085774354 +0000 UTC m=+1004.681591021" watchObservedRunningTime="2026-10-30 21:45:36.088049353 +0000 UTC m=+1004.683866013" Oct 30 21:45:36.482513 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:36.482463 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:37.080413 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:37.080372 5801 generic.go:356] "Generic (PLEG): container finished" podID="db889eca-5e32-4528-b653-c72e065bd1e5" containerID="d1e0e9c07264e74f7b06ba9721ad499bc6796c1114a8fd5e28d34f3ad736d1bf" exitCode=0 Oct 30 21:45:37.080843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:37.080455 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02f8610-g9grl" event={"ID":"db889eca-5e32-4528-b653-c72e065bd1e5","Type":"ContainerDied","Data":"d1e0e9c07264e74f7b06ba9721ad499bc6796c1114a8fd5e28d34f3ad736d1bf"} Oct 30 21:45:37.319454 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:37.319414 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:37.526182 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:37.526093 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2b05c52-7674-4755-8418-96a26244f178" path="/var/lib/kubelet/pods/a2b05c52-7674-4755-8418-96a26244f178/volumes" Oct 30 21:45:38.134718 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:38.134681 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:38.302249 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:38.302216 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02f8610-g9grl" Oct 30 21:45:38.418940 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:38.418854 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db889eca-5e32-4528-b653-c72e065bd1e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db889eca-5e32-4528-b653-c72e065bd1e5-maas-api-xks-ca\") pod \"db889eca-5e32-4528-b653-c72e065bd1e5\" (UID: \"db889eca-5e32-4528-b653-c72e065bd1e5\") " Oct 30 21:45:38.419088 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:38.419003 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db889eca-5e32-4528-b653-c72e065bd1e5-kube-api-access-7qt4d\" (UniqueName: \"kubernetes.io/projected/db889eca-5e32-4528-b653-c72e065bd1e5-kube-api-access-7qt4d\") pod \"db889eca-5e32-4528-b653-c72e065bd1e5\" (UID: \"db889eca-5e32-4528-b653-c72e065bd1e5\") " Oct 30 21:45:38.419217 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:38.419205 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db889eca-5e32-4528-b653-c72e065bd1e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db889eca-5e32-4528-b653-c72e065bd1e5-maas-api-openshift-ca\") pod \"db889eca-5e32-4528-b653-c72e065bd1e5\" (UID: \"db889eca-5e32-4528-b653-c72e065bd1e5\") " Oct 30 21:45:38.419655 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:38.419624 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db889eca-5e32-4528-b653-c72e065bd1e5-maas-api-openshift-ca" pod "db889eca-5e32-4528-b653-c72e065bd1e5" (UID: "db889eca-5e32-4528-b653-c72e065bd1e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:38.419977 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:38.419954 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db889eca-5e32-4528-b653-c72e065bd1e5-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:38.421094 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:38.421069 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db889eca-5e32-4528-b653-c72e065bd1e5-maas-api-xks-ca" pod "db889eca-5e32-4528-b653-c72e065bd1e5" (UID: "db889eca-5e32-4528-b653-c72e065bd1e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:38.421658 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:38.421637 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db889eca-5e32-4528-b653-c72e065bd1e5-kube-api-access-7qt4d" pod "db889eca-5e32-4528-b653-c72e065bd1e5" (UID: "db889eca-5e32-4528-b653-c72e065bd1e5"). InnerVolumeSpecName "kube-api-access-7qt4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:38.522132 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:38.522098 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qt4d\" (UniqueName: \"kubernetes.io/projected/db889eca-5e32-4528-b653-c72e065bd1e5-kube-api-access-7qt4d\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:38.522132 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:38.522130 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db889eca-5e32-4528-b653-c72e065bd1e5-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:38.859416 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:38.859382 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:39.105159 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:39.105127 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02f8610-g9grl" event={"ID":"db889eca-5e32-4528-b653-c72e065bd1e5","Type":"ContainerDied","Data":"4c94c5775c89f4a3bbb86abf230b425c0ce25276a7a4dc3d2e4a8c71fd13ade2"} Oct 30 21:45:39.105159 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:39.105151 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02f8610-g9grl" Oct 30 21:45:39.105359 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:39.105173 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c94c5775c89f4a3bbb86abf230b425c0ce25276a7a4dc3d2e4a8c71fd13ade2" Oct 30 21:45:39.625602 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:39.625570 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:40.319615 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:40.319575 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:41.141374 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:41.141339 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:41.385216 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:41.385180 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:42.031971 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:42.031936 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8eb4f19-7455v"] Oct 30 21:45:42.038052 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:42.038024 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8eb4f19-7455v"] Oct 30 21:45:42.043673 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:42.043644 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:42.732523 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:42.732479 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:43.479598 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:43.479559 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:43.559629 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:43.559589 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0509334-ceac-4802-8bcf-780c4cf295de" path="/var/lib/kubelet/pods/b0509334-ceac-4802-8bcf-780c4cf295de/volumes" Oct 30 21:45:44.325208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:44.325167 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:45.089287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:45.089247 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:45.198224 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:45.198189 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:45.964051 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:45.964009 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:46.731851 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:46.731810 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:47.042880 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:47.042806 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d073a5cf-gxs6l"] Oct 30 21:45:47.048143 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:47.048107 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d073a5cf-gxs6l"] Oct 30 21:45:47.531776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:47.530973 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70ed8e91-3f4f-4fb1-9ab1-abb393956860" path="/var/lib/kubelet/pods/70ed8e91-3f4f-4fb1-9ab1-abb393956860/volumes" Oct 30 21:45:47.634620 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:47.634562 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:48.256913 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:48.256878 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:48.554807 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:48.554771 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:49.390213 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:49.383576 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:50.028863 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:50.028826 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3c9f153-vrtcb"] Oct 30 21:45:50.034348 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:50.034324 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3c9f153-vrtcb"] Oct 30 21:45:50.126972 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:50.126935 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:51.022367 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:51.022326 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:51.527319 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:51.527288 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb" path="/var/lib/kubelet/pods/c4724fdd-3e14-4f34-8da4-ae4d7d2c20cb/volumes" Oct 30 21:45:51.727507 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:51.727473 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:52.367290 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:52.367254 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:52.467245 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:52.467211 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:53.152303 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:53.152270 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:53.832206 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:53.832167 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:54.536122 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:54.536082 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:55.251347 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:55.251293 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:56.077528 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:56.077495 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:56.459183 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:56.459099 5801 scope.go:122] "RemoveContainer" containerID="af013c85e9423c77a066159faf99dffaee20e0ff35b3e1a4fad37cc0a22f356f" Oct 30 21:45:56.473691 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:56.473662 5801 scope.go:122] "RemoveContainer" containerID="00c6d3d478d1a8a93d31113546340998c9854a9582b338de78d071697b0d1418" Oct 30 21:45:56.489324 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:56.489296 5801 scope.go:122] "RemoveContainer" containerID="ab80284f5438f212fa8b5a94e1c7e0fe4e7264e6dadaeb079d88e3c91a6a8241" Oct 30 21:45:56.512361 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:56.512336 5801 scope.go:122] "RemoveContainer" containerID="cba6e040e21aaccf56a9971017446ca8dee473191ae2cc7d4f08dda67f1b67c1" Oct 30 21:45:56.526407 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:56.526376 5801 scope.go:122] "RemoveContainer" containerID="659824c8255b925dd92d9f7974d2532ca8d8c3a6eb918b36626c347ede86fd75" Oct 30 21:45:56.537813 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:56.537790 5801 scope.go:122] "RemoveContainer" containerID="832c16801e187f8e5eab21032594f4ec3b416e10ab0c759d97bf22417d9066f5" Oct 30 21:45:56.547951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:56.547929 5801 scope.go:122] "RemoveContainer" containerID="6fb385f66bcf153f8fa461795f5eea0abbc26a8931d7fd8b94501aa42eb33b2b" Oct 30 21:45:56.558095 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:56.558072 5801 scope.go:122] "RemoveContainer" containerID="be99c0dd97261916fb151bd94d27f8448ebb2d7bc996ae7b00f069f75c6c8fe6" Oct 30 21:45:56.568525 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:56.568503 5801 scope.go:122] "RemoveContainer" containerID="49605a76a8ac69fff2e2179c87e7a42f9cc11788246cd8ef4377083b23dd2dcb" Oct 30 21:45:56.578200 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:56.578183 5801 scope.go:122] "RemoveContainer" containerID="97ef059f678c1a11cb8d5e575356942672a321d56bf0a2b659ae6d14be6382bb" Oct 30 21:45:56.588105 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:56.588089 5801 scope.go:122] "RemoveContainer" containerID="519b35bc928c60457e4d5062c94a1d7c82f31ad776a56b508c1bd1701c1bce57" Oct 30 21:45:56.597723 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:56.597706 5801 scope.go:122] "RemoveContainer" containerID="ed83303962d68fdfd6ab9706012393cc179e30eb363a8cf55edab64c55343355" Oct 30 21:45:56.607546 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:56.607518 5801 scope.go:122] "RemoveContainer" containerID="8b6eb94a7b8444c353c1dd66b05caba0226aad28378f670c36cd83672a73a0d2" Oct 30 21:45:56.620193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:56.620160 5801 scope.go:122] "RemoveContainer" containerID="edad792c0da173d2d283ed802512797603dac64c33e8e5f2e0d5e07b869dfd1b" Oct 30 21:45:56.631792 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:56.631738 5801 scope.go:122] "RemoveContainer" containerID="022207ac1bf43d748cfba53ba9a53be8cbb35e9fb042611963bba2819e94c7cd" Oct 30 21:45:56.641464 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:56.641444 5801 scope.go:122] "RemoveContainer" containerID="32ca5e686e231e1a3f923052a2e121a890028b21d5dc9c20db9f75a2db66af9b" Oct 30 21:45:56.919716 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:56.919635 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:57.803255 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:57.803210 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:58.023801 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:58.020098 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:59.135320 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:45:59.135279 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:00.215905 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:00.215860 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:00.488828 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:46:00.486526 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:46:00.496779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:00.494713 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:00.520777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:00.519190 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:00.520777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:00.519604 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 21:46:00.607789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:00.607654 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:00.628778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:00.623827 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:00.628778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:00.624359 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:46:00.628778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:00.624802 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:46:00.628778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:00.625210 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:46:01.042089 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:01.041836 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:01.248819 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:01.243258 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:01.258856 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:01.258822 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:01.497770 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:01.497201 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:01.499348 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:01.498396 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:46:01.525302 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:01.524215 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:01.525302 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:01.524561 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:46:01.827887 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:01.721816 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:46:01.827887 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:01.722163 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:46:01.827887 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:01.722419 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:46:01.827887 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:01.744795 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:01.827887 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:01.747251 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:46:01.827887 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:01.792837 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:46:01.827887 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:01.792896 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" podUID="9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:01.924363 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:01.850877 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:46:01.924363 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:01.850940 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" podUID="493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:02.001069 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.001037 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:46:02.001377 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.001345 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:46:02.001549 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.001537 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:46:02.001727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.001704 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:46:02.164853 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.066360 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:46:02.164853 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.066416 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" podUID="fda6dedb-779c-49d8-8fcf-ba31f73402ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:02.208833 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.208796 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:46:02.209268 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.209158 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:46:02.209268 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.209212 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:46:02.250775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.250715 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:02.281003 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.266801 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:02.295808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.266852 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:02.296233 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.296199 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:46:02.317225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.312931 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:02.331894 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.331847 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:46:02.379744 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.371140 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:02.396666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.395669 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:46:02.412821 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.412790 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:46:02.448534 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.448455 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:46:02.630771 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.458089 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:02.630771 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.504291 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:46:02.630771 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.545601 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:46:02.691772 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.690583 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:46:02.741045 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.715722 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:46:02.741045 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.717535 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:46:02.741045 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.722393 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:02.741045 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.735117 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:46:02.791784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.776428 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:46:02.819056 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.819027 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:46:02.890845 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:02.890810 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:03.033435 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:03.033343 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b85feb-8xhgm"] Oct 30 21:46:03.038701 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:03.038667 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b85feb-8xhgm"] Oct 30 21:46:03.525664 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:03.525634 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88539ee2-49d3-4d50-8903-928974b64c30" path="/var/lib/kubelet/pods/88539ee2-49d3-4d50-8903-928974b64c30/volumes" Oct 30 21:46:03.741986 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:03.741949 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:04.036885 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:04.036844 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66c1025-gv9zf"] Oct 30 21:46:04.041423 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:04.041395 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66c1025-gv9zf"] Oct 30 21:46:04.521093 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:04.521045 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:05.036677 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:05.036632 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81f14d1c-5765f"] Oct 30 21:46:05.042300 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:05.042269 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d8d6034-8c89n"] Oct 30 21:46:05.062202 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:05.062168 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81f14d1c-5765f"] Oct 30 21:46:05.067001 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:05.066978 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d8d6034-8c89n"] Oct 30 21:46:05.436793 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:05.436721 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:05.541117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:05.541069 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d7eb05e-10e3-42ae-ae42-9c7426701314" path="/var/lib/kubelet/pods/1d7eb05e-10e3-42ae-ae42-9c7426701314/volumes" Oct 30 21:46:05.542292 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:05.542263 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ab3247a-845f-4b65-98bb-fd9752468cf6" path="/var/lib/kubelet/pods/6ab3247a-845f-4b65-98bb-fd9752468cf6/volumes" Oct 30 21:46:05.542914 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:05.542891 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff86e576-1804-4310-aee0-cc050227805d" path="/var/lib/kubelet/pods/ff86e576-1804-4310-aee0-cc050227805d/volumes" Oct 30 21:46:06.262463 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:06.262422 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:07.027840 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:07.027806 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:07.775638 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:07.775557 5801 generic.go:356] "Generic (PLEG): container finished" podID="457b1dc9-f94b-4dda-8b46-e125e0069033" containerID="0046d3214c1887638295980b75d32b3ee903884b4ca8bbcb13b97e894e8aca52" exitCode=0 Oct 30 21:46:07.775638 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:07.775621 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" event={"ID":"457b1dc9-f94b-4dda-8b46-e125e0069033","Type":"ContainerDied","Data":"0046d3214c1887638295980b75d32b3ee903884b4ca8bbcb13b97e894e8aca52"} Oct 30 21:46:07.776164 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:07.776070 5801 scope.go:122] "RemoveContainer" containerID="0046d3214c1887638295980b75d32b3ee903884b4ca8bbcb13b97e894e8aca52" Oct 30 21:46:07.777969 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:07.777939 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:07.778091 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:07.778026 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:07.778091 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:07.778074 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:07.815034 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:07.811006 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:08.572959 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:08.572925 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:08.794653 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:08.794611 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" event={"ID":"457b1dc9-f94b-4dda-8b46-e125e0069033","Type":"ContainerStarted","Data":"1327543b60224dc1bd2af1573c0220b80df821174c3378522963597b4e7fa436"} Oct 30 21:46:08.974839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:08.973851 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:46:09.371776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:09.371144 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:09.775835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:09.768970 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:46:09.775835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:09.769116 5801 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 30 21:46:09.775835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:09.769161 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" podUID="457b1dc9-f94b-4dda-8b46-e125e0069033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 30 21:46:10.120780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:10.118880 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:10.727592 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:10.727532 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:10.768688 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:10.768111 5801 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 30 21:46:10.768688 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:10.768179 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" podUID="457b1dc9-f94b-4dda-8b46-e125e0069033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 30 21:46:10.979143 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:10.979053 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:11.030181 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.030151 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2eda136-jh2tv"] Oct 30 21:46:11.035372 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.035347 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2eda136-jh2tv"] Oct 30 21:46:11.398419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.398382 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb"] Oct 30 21:46:11.399591 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.399553 5801 cpu_manager.go:414] "containerMap: removing container" podUID="db889eca-5e32-4528-b653-c72e065bd1e5" containerName="revoke-keys" Oct 30 21:46:11.399591 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.399579 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db889eca-5e32-4528-b653-c72e065bd1e5" containerName="revoke-keys" Oct 30 21:46:11.399851 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.399778 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="db889eca-5e32-4528-b653-c72e065bd1e5" containerName="revoke-keys" Oct 30 21:46:11.405322 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.405298 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.416288 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.414336 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7386906d-openshift-default-dockercfg-ffr97\"" Oct 30 21:46:11.420284 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.420259 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb"] Oct 30 21:46:11.438073 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.438046 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:11.485491 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.485447 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-data\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.485686 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.485571 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-credential-socket\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.485686 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.485620 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-workload-socket\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.485920 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.485746 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istiod-ca-cert\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.485920 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.485807 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-podinfo\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.485920 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.485860 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-token\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.486078 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.486014 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-envoy\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.486172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.486150 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nrwz\" (UniqueName: \"kubernetes.io/projected/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-kube-api-access-4nrwz\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.486311 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.486183 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-workload-certs\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.527101 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.527069 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7836e324-455e-47b4-96df-586ef9cfd53e" path="/var/lib/kubelet/pods/7836e324-455e-47b4-96df-586ef9cfd53e/volumes" Oct 30 21:46:11.537008 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.536975 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:11.618777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.613971 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nrwz\" (UniqueName: \"kubernetes.io/projected/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-kube-api-access-4nrwz\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.618777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.614031 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-workload-certs\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.618777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.614214 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-data\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.618777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.614262 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-credential-socket\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.618777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.614310 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-workload-socket\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.618777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.614397 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istiod-ca-cert\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.618777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.614439 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-podinfo\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.618777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.614476 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-token\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.618777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.614573 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-envoy\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.618777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.617318 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-credential-socket\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.618777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.617913 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-workload-certs\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.618777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.618159 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-data\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.618777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.618388 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-workload-socket\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.618777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.618464 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istiod-ca-cert\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.644279 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.643260 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-podinfo\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.647816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.647792 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-token\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.661349 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.661272 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:11.689018 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.678376 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-envoy\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.691549 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.691522 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nrwz\" (UniqueName: \"kubernetes.io/projected/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-kube-api-access-4nrwz\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.739776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.739042 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:11.835744 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.835703 5801 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 30 21:46:11.885781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:11.877863 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" podUID="457b1dc9-f94b-4dda-8b46-e125e0069033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 30 21:46:12.308345 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.308305 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:12.413483 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.411417 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5"] Oct 30 21:46:12.420477 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.420450 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5"] Oct 30 21:46:12.421847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.420991 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.433464 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.433418 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-6cf5f0-openshift-default-dockercfg-4qhw8\"" Oct 30 21:46:12.488961 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.488785 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:12.555319 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.552444 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-token\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.555319 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.552520 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-workload-socket\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.555319 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.552599 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-envoy\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.555319 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.552658 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-data\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.555319 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.552684 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-podinfo\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.555319 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.552730 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-credential-socket\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.555319 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.552881 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfvtd\" (UniqueName: \"kubernetes.io/projected/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-kube-api-access-nfvtd\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.555319 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.552981 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istiod-ca-cert\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.555319 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.553224 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-workload-certs\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.572895 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.572804 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb"] Oct 30 21:46:12.600402 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.600365 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:12.600604 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.600584 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:12.600742 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.600726 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:12.673783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.657953 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-workload-socket\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.685797 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.672477 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-workload-socket\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.686094 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.686066 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-envoy\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.687867 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.687840 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-data\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.688043 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.687066 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-data\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.688915 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.688890 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-podinfo\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.689547 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.689526 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-credential-socket\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.690238 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.690216 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfvtd\" (UniqueName: \"kubernetes.io/projected/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-kube-api-access-nfvtd\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.691790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.690945 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istiod-ca-cert\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.691790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.691138 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-workload-certs\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.691790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.691187 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-token\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.691790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.690020 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-credential-socket\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.691790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.691704 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-workload-certs\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.692032 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.691863 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istiod-ca-cert\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.692256 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.692228 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-podinfo\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.692358 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.692278 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-envoy\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.703194 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.703168 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfvtd\" (UniqueName: \"kubernetes.io/projected/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-kube-api-access-nfvtd\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.704257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.704214 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-token\") pod \"e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.751494 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.751464 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:12.775775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.775062 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:12.779774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.776691 5801 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 30 21:46:12.779774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:12.776783 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" podUID="457b1dc9-f94b-4dda-8b46-e125e0069033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 30 21:46:13.052745 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:13.052606 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" event={"ID":"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf","Type":"ContainerStarted","Data":"f57ffb3b59ac5ad6024b8b8e95001f7fabea9c4c10c94b3787e929cac75b5b2e"} Oct 30 21:46:13.052745 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:13.052689 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" event={"ID":"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf","Type":"ContainerStarted","Data":"a47bff197fe163398037b914aa180772d96aeb774f95e4a35131e19c8279e9a7"} Oct 30 21:46:13.141912 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:13.140769 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" podStartSLOduration=2.140728952 podStartE2EDuration="2.140728952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:13.136337511 +0000 UTC m=+1041.732154170" watchObservedRunningTime="2026-10-30 21:46:13.140728952 +0000 UTC m=+1041.736545614" Oct 30 21:46:13.300784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:13.291874 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:13.331778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:13.327810 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5"] Oct 30 21:46:13.373330 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:13.372080 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:13.373330 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:13.372179 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:13.373330 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:13.372224 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:13.740649 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:13.740565 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:13.746476 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:13.746451 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:13.775776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:13.774140 5801 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 30 21:46:13.775776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:13.774200 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" podUID="457b1dc9-f94b-4dda-8b46-e125e0069033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 30 21:46:14.090468 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:14.090433 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" event={"ID":"a7e5545a-7bc6-437b-bef6-4c6ee6776f09","Type":"ContainerStarted","Data":"d944e9bb4c96c7dcf933f124619b6d599bee4f73f83a6c6712a109b6f60cfb72"} Oct 30 21:46:14.090698 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:14.090488 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" event={"ID":"a7e5545a-7bc6-437b-bef6-4c6ee6776f09","Type":"ContainerStarted","Data":"bab9ccb8e3e498268d8e8fb577678f17ce3d0a9973a6ce19ea8510ce2b808941"} Oct 30 21:46:14.090852 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:14.090832 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:14.091877 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:14.091856 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:46:14.101697 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:14.101667 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:14.121020 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:14.120976 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" podStartSLOduration=2.120963895 podStartE2EDuration="2.120963895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:14.113644108 +0000 UTC m=+1042.709460760" watchObservedRunningTime="2026-10-30 21:46:14.120963895 +0000 UTC m=+1042.716780555" Oct 30 21:46:14.770533 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:14.770495 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:46:14.771641 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:14.771617 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:46:14.776178 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:14.776156 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:14.781173 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:14.781153 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:15.100450 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:15.100414 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:15.100703 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:15.100672 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:46:15.137232 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:15.137197 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:15.642789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:15.642719 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:15.745589 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:15.745556 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:16.146798 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:16.146768 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:16.430378 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:16.430301 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:16.536289 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:16.536253 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:16.960092 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:16.960056 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:17.017812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.017765 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq"] Oct 30 21:46:17.022415 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.022391 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq" Oct 30 21:46:17.032080 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.032055 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7386906d\"" Oct 30 21:46:17.047685 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.047651 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq"] Oct 30 21:46:17.056667 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.056638 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7386906d-78c5b4547c-84v44"] Oct 30 21:46:17.062625 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.062603 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7386906d-78c5b4547c-84v44" Oct 30 21:46:17.070482 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.070454 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7386906d-dockercfg-tn5zh\"" Oct 30 21:46:17.075368 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.075319 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7386906d-78c5b4547c-84v44"] Oct 30 21:46:17.129552 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.129507 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:17.153795 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.153723 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30476067-bd3b-4eac-825e-19fc7879eed8-plugins-config-volume\") pod \"payload-processing-e2e-mt-7386906d-78c5b4547c-84v44\" (UID: \"30476067-bd3b-4eac-825e-19fc7879eed8\") " pod="openshift-ingress/payload-processing-e2e-mt-7386906d-78c5b4547c-84v44" Oct 30 21:46:17.154223 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.153974 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76f0a594-7906-4abe-bb8f-a4265394e9ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq\" (UID: \"76f0a594-7906-4abe-bb8f-a4265394e9ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq" Oct 30 21:46:17.154223 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.154109 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t44j9\" (UniqueName: \"kubernetes.io/projected/76f0a594-7906-4abe-bb8f-a4265394e9ce-kube-api-access-t44j9\") pod \"payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq\" (UID: \"76f0a594-7906-4abe-bb8f-a4265394e9ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq" Oct 30 21:46:17.154329 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.154244 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdvjm\" (UniqueName: \"kubernetes.io/projected/30476067-bd3b-4eac-825e-19fc7879eed8-kube-api-access-jdvjm\") pod \"payload-processing-e2e-mt-7386906d-78c5b4547c-84v44\" (UID: \"30476067-bd3b-4eac-825e-19fc7879eed8\") " pod="openshift-ingress/payload-processing-e2e-mt-7386906d-78c5b4547c-84v44" Oct 30 21:46:17.256836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.255981 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdvjm\" (UniqueName: \"kubernetes.io/projected/30476067-bd3b-4eac-825e-19fc7879eed8-kube-api-access-jdvjm\") pod \"payload-processing-e2e-mt-7386906d-78c5b4547c-84v44\" (UID: \"30476067-bd3b-4eac-825e-19fc7879eed8\") " pod="openshift-ingress/payload-processing-e2e-mt-7386906d-78c5b4547c-84v44" Oct 30 21:46:17.256836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.256244 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30476067-bd3b-4eac-825e-19fc7879eed8-plugins-config-volume\") pod \"payload-processing-e2e-mt-7386906d-78c5b4547c-84v44\" (UID: \"30476067-bd3b-4eac-825e-19fc7879eed8\") " pod="openshift-ingress/payload-processing-e2e-mt-7386906d-78c5b4547c-84v44" Oct 30 21:46:17.256836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.256407 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76f0a594-7906-4abe-bb8f-a4265394e9ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq\" (UID: \"76f0a594-7906-4abe-bb8f-a4265394e9ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq" Oct 30 21:46:17.256836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.256484 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t44j9\" (UniqueName: \"kubernetes.io/projected/76f0a594-7906-4abe-bb8f-a4265394e9ce-kube-api-access-t44j9\") pod \"payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq\" (UID: \"76f0a594-7906-4abe-bb8f-a4265394e9ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq" Oct 30 21:46:17.266101 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.258629 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30476067-bd3b-4eac-825e-19fc7879eed8-plugins-config-volume\") pod \"payload-processing-e2e-mt-7386906d-78c5b4547c-84v44\" (UID: \"30476067-bd3b-4eac-825e-19fc7879eed8\") " pod="openshift-ingress/payload-processing-e2e-mt-7386906d-78c5b4547c-84v44" Oct 30 21:46:17.266101 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.260578 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76f0a594-7906-4abe-bb8f-a4265394e9ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq\" (UID: \"76f0a594-7906-4abe-bb8f-a4265394e9ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq" Oct 30 21:46:17.316793 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.298350 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdvjm\" (UniqueName: \"kubernetes.io/projected/30476067-bd3b-4eac-825e-19fc7879eed8-kube-api-access-jdvjm\") pod \"payload-processing-e2e-mt-7386906d-78c5b4547c-84v44\" (UID: \"30476067-bd3b-4eac-825e-19fc7879eed8\") " pod="openshift-ingress/payload-processing-e2e-mt-7386906d-78c5b4547c-84v44" Oct 30 21:46:17.316793 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.304643 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t44j9\" (UniqueName: \"kubernetes.io/projected/76f0a594-7906-4abe-bb8f-a4265394e9ce-kube-api-access-t44j9\") pod \"payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq\" (UID: \"76f0a594-7906-4abe-bb8f-a4265394e9ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq" Oct 30 21:46:17.375777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.374727 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq" Oct 30 21:46:17.394286 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.387322 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7386906d-78c5b4547c-84v44" Oct 30 21:46:17.770615 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.770563 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq"] Oct 30 21:46:17.771450 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:46:17.771424 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f0a594_7906_4abe_bb8f_a4265394e9ce.slice/crio-9a89ba98dd1a9fac16136b7010ccb600d87f597a7ae6423eb73fb941d5d1979f WatchSource:0}: Error finding container 9a89ba98dd1a9fac16136b7010ccb600d87f597a7ae6423eb73fb941d5d1979f: Status 404 returned error can't find the container with id 9a89ba98dd1a9fac16136b7010ccb600d87f597a7ae6423eb73fb941d5d1979f Oct 30 21:46:17.780008 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.779849 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:17.846485 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.846459 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5"] Oct 30 21:46:17.851136 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.851119 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5" Oct 30 21:46:17.862105 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.860635 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-6cf5f0-dockercfg-wtstc\"" Oct 30 21:46:17.862105 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.861039 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-6cf5f0\"" Oct 30 21:46:17.863391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.862740 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5"] Oct 30 21:46:17.869967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.869941 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7386906d-78c5b4547c-84v44"] Oct 30 21:46:17.888343 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.887156 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:17.928005 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.927971 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gzqp\" (UniqueName: \"kubernetes.io/projected/c6a121f3-14a8-447c-bdad-ac188d32830c-kube-api-access-8gzqp\") pod \"payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5\" (UID: \"c6a121f3-14a8-447c-bdad-ac188d32830c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5" Oct 30 21:46:17.928157 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.928134 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a121f3-14a8-447c-bdad-ac188d32830c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5\" (UID: \"c6a121f3-14a8-447c-bdad-ac188d32830c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5" Oct 30 21:46:17.987030 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:17.986999 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:18.047448 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:18.046008 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a121f3-14a8-447c-bdad-ac188d32830c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5\" (UID: \"c6a121f3-14a8-447c-bdad-ac188d32830c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5" Oct 30 21:46:18.047448 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:18.046447 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gzqp\" (UniqueName: \"kubernetes.io/projected/c6a121f3-14a8-447c-bdad-ac188d32830c-kube-api-access-8gzqp\") pod \"payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5\" (UID: \"c6a121f3-14a8-447c-bdad-ac188d32830c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5" Oct 30 21:46:18.047851 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:18.047820 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a121f3-14a8-447c-bdad-ac188d32830c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5\" (UID: \"c6a121f3-14a8-447c-bdad-ac188d32830c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5" Oct 30 21:46:18.084777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:18.084006 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gzqp\" (UniqueName: \"kubernetes.io/projected/c6a121f3-14a8-447c-bdad-ac188d32830c-kube-api-access-8gzqp\") pod \"payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5\" (UID: \"c6a121f3-14a8-447c-bdad-ac188d32830c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5" Oct 30 21:46:18.165776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:18.155128 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7386906d-78c5b4547c-84v44" event={"ID":"30476067-bd3b-4eac-825e-19fc7879eed8","Type":"ContainerStarted","Data":"e75b09e74f8898948f607b64f1b3b66ab2f503907245464f8043cbfb65c448f8"} Oct 30 21:46:18.165776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:18.155195 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7386906d-78c5b4547c-84v44" event={"ID":"30476067-bd3b-4eac-825e-19fc7879eed8","Type":"ContainerStarted","Data":"78fc2ae9689f788b940045d845a58036d5d9cfdd0520b402bd9f145dad136c4f"} Oct 30 21:46:18.165776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:18.156445 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7386906d-78c5b4547c-84v44" Oct 30 21:46:18.192774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:18.173938 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq" event={"ID":"76f0a594-7906-4abe-bb8f-a4265394e9ce","Type":"ContainerStarted","Data":"fb38dcf70dcba431793c7546ef8dd7c92f4ff9795af0e4400ad25ffb9b3be367"} Oct 30 21:46:18.192774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:18.174006 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq" event={"ID":"76f0a594-7906-4abe-bb8f-a4265394e9ce","Type":"ContainerStarted","Data":"9a89ba98dd1a9fac16136b7010ccb600d87f597a7ae6423eb73fb941d5d1979f"} Oct 30 21:46:18.192774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:18.174035 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq" Oct 30 21:46:18.192774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:18.174270 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5" Oct 30 21:46:18.251775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:18.248098 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7386906d-78c5b4547c-84v44" podStartSLOduration=1.2480793079999999 podStartE2EDuration="1.248079308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:18.209110867 +0000 UTC m=+1046.804927540" watchObservedRunningTime="2026-10-30 21:46:18.248079308 +0000 UTC m=+1046.843895969" Oct 30 21:46:18.589295 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:18.589233 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq" podStartSLOduration=2.58921293 podStartE2EDuration="2.58921293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:18.298557141 +0000 UTC m=+1046.894373863" watchObservedRunningTime="2026-10-30 21:46:18.58921293 +0000 UTC m=+1047.185029592" Oct 30 21:46:18.590583 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:46:18.590554 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a121f3_14a8_447c_bdad_ac188d32830c.slice/crio-aec8c7472c9ee95788f795c8681b86c5c634fe49bda4dd16cb1875154f91a98b WatchSource:0}: Error finding container aec8c7472c9ee95788f795c8681b86c5c634fe49bda4dd16cb1875154f91a98b: Status 404 returned error can't find the container with id aec8c7472c9ee95788f795c8681b86c5c634fe49bda4dd16cb1875154f91a98b Oct 30 21:46:18.593028 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:18.593003 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5"] Oct 30 21:46:18.941990 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:18.941899 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:19.187773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:19.187718 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5" event={"ID":"c6a121f3-14a8-447c-bdad-ac188d32830c","Type":"ContainerStarted","Data":"e8ee73657ede5796664e11bcaa0e746d993c0a931a5aed027108dab27394a2cd"} Oct 30 21:46:19.188195 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:19.187803 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5" event={"ID":"c6a121f3-14a8-447c-bdad-ac188d32830c","Type":"ContainerStarted","Data":"aec8c7472c9ee95788f795c8681b86c5c634fe49bda4dd16cb1875154f91a98b"} Oct 30 21:46:19.207404 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:19.207317 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5" podStartSLOduration=2.207303398 podStartE2EDuration="2.207303398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:19.206064535 +0000 UTC m=+1047.801881210" watchObservedRunningTime="2026-10-30 21:46:19.207303398 +0000 UTC m=+1047.803120086" Oct 30 21:46:19.650780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:19.650706 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:20.027917 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:20.027832 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33edf72f-ddl6r"] Oct 30 21:46:20.045541 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:20.045385 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33edf72f-ddl6r"] Oct 30 21:46:20.198534 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:20.198495 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5" Oct 30 21:46:20.373741 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:20.373703 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:21.052490 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:21.052443 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:21.528080 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:21.528044 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="314732a1-75c0-4928-8467-9a3bc84729ff" path="/var/lib/kubelet/pods/314732a1-75c0-4928-8467-9a3bc84729ff/volumes" Oct 30 21:46:21.824496 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:21.824453 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:22.495315 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:22.495280 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:23.239392 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:23.239359 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:23.949795 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:23.949730 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:24.645764 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:24.645716 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:25.331843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:25.331797 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:26.031542 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:26.031499 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:26.759177 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:26.759146 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:26.868774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:26.868671 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:27.056867 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:27.056833 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:27.740398 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:27.740359 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:28.619039 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:28.619001 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:28.724049 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:28.723849 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:28.834175 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:28.834137 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:29.224995 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:29.224966 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq" Oct 30 21:46:29.449738 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:29.449702 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:30.161281 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:30.161243 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:30.219304 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:30.219272 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7386906d-78c5b4547c-84v44" Oct 30 21:46:30.319038 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:46:30.318991 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:46:30.885122 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:30.885082 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:31.216810 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:31.216703 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5" Oct 30 21:46:31.840910 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:31.840868 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:32.557487 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:32.557453 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:33.033605 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:33.033524 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1caf5a9b-ndfhz"] Oct 30 21:46:33.042449 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:33.042421 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1caf5a9b-ndfhz"] Oct 30 21:46:33.237871 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:33.237833 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:33.527124 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:33.527093 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1485533-fd44-4182-a51a-8ed758b41442" path="/var/lib/kubelet/pods/b1485533-fd44-4182-a51a-8ed758b41442/volumes" Oct 30 21:46:33.957091 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:33.957058 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:34.704914 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:34.704875 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:35.445510 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:35.445471 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:36.120304 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:36.120265 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:36.861979 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:36.861943 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:37.755698 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:37.755665 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:38.575143 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:38.575111 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:39.371312 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:39.371282 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:40.250672 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:40.250634 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:41.065471 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:41.065434 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:41.079881 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:41.079836 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c701fda2-f82w4"] Oct 30 21:46:41.103658 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:41.103615 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c701fda2-f82w4"] Oct 30 21:46:41.527786 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:41.527675 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd5a6ffe-c3ea-4868-9ac5-6ed43040349e" path="/var/lib/kubelet/pods/fd5a6ffe-c3ea-4868-9ac5-6ed43040349e/volumes" Oct 30 21:46:41.988182 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:41.988144 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:42.946742 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:42.946690 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:43.033830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:43.033798 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ac40742-qndwc"] Oct 30 21:46:43.042573 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:43.042533 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ac40742-qndwc"] Oct 30 21:46:43.533471 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:43.533431 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09502484-dcf7-4af4-a220-628ef74cb996" path="/var/lib/kubelet/pods/09502484-dcf7-4af4-a220-628ef74cb996/volumes" Oct 30 21:46:43.748082 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:43.748042 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:44.429319 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:44.429280 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:45.303712 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:45.303666 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:46.007998 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:46.007946 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:46.112889 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:46.112855 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:46.765466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:46.765432 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:46.976339 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:46.976288 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:47.069708 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:47.069673 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d59c412-mhbzr"] Oct 30 21:46:47.089688 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:47.089639 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d59c412-mhbzr"] Oct 30 21:46:47.218776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:47.214243 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:47.491136 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:47.491055 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:47.530257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:47.530223 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a079e35b-efc4-4cda-9cff-fcb6270ad56a" path="/var/lib/kubelet/pods/a079e35b-efc4-4cda-9cff-fcb6270ad56a/volumes" Oct 30 21:46:47.666802 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:47.666729 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:47.896705 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:47.896662 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:47.941685 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:47.941649 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz"] Oct 30 21:46:47.947847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:47.947821 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:47.955632 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:47.955605 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-aad4dd-openshift-default-dockercfg-9x45c\"" Oct 30 21:46:47.963404 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:47.963373 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz"] Oct 30 21:46:48.016973 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.016932 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:48.124529 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.124491 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:48.141794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.140202 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-envoy\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.141794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.140435 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-workload-socket\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.141794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.140592 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-token\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.141794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.140654 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-podinfo\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.141794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.140697 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-data\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.141794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.140789 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istiod-ca-cert\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.141794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.140823 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-credential-socket\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.141794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.140878 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-workload-certs\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.141794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.140953 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcrqd\" (UniqueName: \"kubernetes.io/projected/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-kube-api-access-wcrqd\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.243529 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.243431 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istiod-ca-cert\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.253877 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.244503 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istiod-ca-cert\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.254631 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.254600 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-credential-socket\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.254842 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.244588 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-credential-socket\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.254992 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.254977 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-workload-certs\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.255483 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.255463 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcrqd\" (UniqueName: \"kubernetes.io/projected/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-kube-api-access-wcrqd\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.255661 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.255643 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-envoy\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.272480 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.255359 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-workload-certs\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.287992 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.272914 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-workload-socket\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.289057 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.289028 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-token\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.289291 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.289271 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-podinfo\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.289428 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.289413 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-data\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.289992 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.289966 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-data\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.290149 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.274109 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-workload-socket\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.305416 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.302584 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcrqd\" (UniqueName: \"kubernetes.io/projected/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-kube-api-access-wcrqd\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.311208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.310963 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-envoy\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.328450 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.328418 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-podinfo\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.357969 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.351913 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-token\") pod \"e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.571503 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.571467 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:48.646592 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.646426 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:48.872802 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:46:48.872734 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb1fb25_0a54_4f2e_9bcb_88e2ebcaef2e.slice/crio-da0ef261e8a27f44a6e8a5ed165cf5b8a6be825eb998172bbe7e14111dcf7bd9 WatchSource:0}: Error finding container da0ef261e8a27f44a6e8a5ed165cf5b8a6be825eb998172bbe7e14111dcf7bd9: Status 404 returned error can't find the container with id da0ef261e8a27f44a6e8a5ed165cf5b8a6be825eb998172bbe7e14111dcf7bd9 Oct 30 21:46:48.874576 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.874458 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz"] Oct 30 21:46:48.875737 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.875707 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:48.875838 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.875809 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:48.875880 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:48.875847 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:49.033513 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:49.033480 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-105c3a96-dqjrf"] Oct 30 21:46:49.039630 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:49.039601 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-105c3a96-dqjrf"] Oct 30 21:46:49.337739 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:49.337696 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:49.530147 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:49.530113 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b156454b-be9d-43f3-9322-dc78aed291b9" path="/var/lib/kubelet/pods/b156454b-be9d-43f3-9322-dc78aed291b9/volumes" Oct 30 21:46:49.578029 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:49.577991 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" event={"ID":"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e","Type":"ContainerStarted","Data":"cbab4eddbdbb21198e1d111ef24547a637bb3ca3648360505872d2cb402dffe9"} Oct 30 21:46:49.578223 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:49.578049 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" event={"ID":"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e","Type":"ContainerStarted","Data":"da0ef261e8a27f44a6e8a5ed165cf5b8a6be825eb998172bbe7e14111dcf7bd9"} Oct 30 21:46:49.606983 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:49.606887 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" podStartSLOduration=2.606870191 podStartE2EDuration="2.606870191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:49.603451236 +0000 UTC m=+1078.199267921" watchObservedRunningTime="2026-10-30 21:46:49.606870191 +0000 UTC m=+1078.202686850" Oct 30 21:46:50.062818 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:50.062784 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:50.572138 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:50.572096 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:50.578497 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:50.578469 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:50.587004 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:50.586979 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:50.587860 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:50.587844 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:46:50.750391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:50.750355 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:51.029018 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:51.028934 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5706d53c-t2bst"] Oct 30 21:46:51.034234 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:51.034208 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5706d53c-t2bst"] Oct 30 21:46:51.526838 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:51.526805 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="701eda53-9940-4dd3-a0db-622c157347e7" path="/var/lib/kubelet/pods/701eda53-9940-4dd3-a0db-622c157347e7/volumes" Oct 30 21:46:51.534139 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:51.534096 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:51.974244 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:51.974202 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:52.163436 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:52.163403 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:52.384902 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:52.384869 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:53.242044 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:53.242007 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:53.347345 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:53.347303 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:53.397206 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:53.397176 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh"] Oct 30 21:46:53.403347 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:53.403327 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh" Oct 30 21:46:53.467862 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:53.459220 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-aad4dd-dockercfg-mbfwc\"" Oct 30 21:46:53.506833 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:53.479722 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-aad4dd\"" Oct 30 21:46:53.764869 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:53.764781 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54e288ae-db61-46b6-884e-0c199fbacaa7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh\" (UID: \"54e288ae-db61-46b6-884e-0c199fbacaa7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh" Oct 30 21:46:53.774199 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:53.774156 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bsjx\" (UniqueName: \"kubernetes.io/projected/54e288ae-db61-46b6-884e-0c199fbacaa7-kube-api-access-9bsjx\") pod \"payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh\" (UID: \"54e288ae-db61-46b6-884e-0c199fbacaa7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh" Oct 30 21:46:53.806558 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:53.806524 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh"] Oct 30 21:46:53.876048 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:53.876013 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bsjx\" (UniqueName: \"kubernetes.io/projected/54e288ae-db61-46b6-884e-0c199fbacaa7-kube-api-access-9bsjx\") pod \"payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh\" (UID: \"54e288ae-db61-46b6-884e-0c199fbacaa7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh" Oct 30 21:46:53.876340 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:53.876317 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54e288ae-db61-46b6-884e-0c199fbacaa7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh\" (UID: \"54e288ae-db61-46b6-884e-0c199fbacaa7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh" Oct 30 21:46:53.876997 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:53.876975 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54e288ae-db61-46b6-884e-0c199fbacaa7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh\" (UID: \"54e288ae-db61-46b6-884e-0c199fbacaa7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh" Oct 30 21:46:53.888920 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:53.888894 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bsjx\" (UniqueName: \"kubernetes.io/projected/54e288ae-db61-46b6-884e-0c199fbacaa7-kube-api-access-9bsjx\") pod \"payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh\" (UID: \"54e288ae-db61-46b6-884e-0c199fbacaa7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh" Oct 30 21:46:53.975894 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:53.975864 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh" Oct 30 21:46:54.033130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:54.032502 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d987ad-tt9nb"] Oct 30 21:46:54.039947 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:54.039924 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d987ad-tt9nb"] Oct 30 21:46:54.258223 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:54.258189 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:54.261569 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:46:54.261541 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e288ae_db61_46b6_884e_0c199fbacaa7.slice/crio-37f0a21586b7039bc7ad0672c845bfd78ed85557c8a348832a766639bf44cc70 WatchSource:0}: Error finding container 37f0a21586b7039bc7ad0672c845bfd78ed85557c8a348832a766639bf44cc70: Status 404 returned error can't find the container with id 37f0a21586b7039bc7ad0672c845bfd78ed85557c8a348832a766639bf44cc70 Oct 30 21:46:54.262550 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:54.262526 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh"] Oct 30 21:46:54.801703 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:54.801616 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh" event={"ID":"54e288ae-db61-46b6-884e-0c199fbacaa7","Type":"ContainerStarted","Data":"cfe52991760ad0ea2cf18555ef13ec3495808680fbff4ce650a67d0642780fac"} Oct 30 21:46:54.801703 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:54.801672 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh" event={"ID":"54e288ae-db61-46b6-884e-0c199fbacaa7","Type":"ContainerStarted","Data":"37f0a21586b7039bc7ad0672c845bfd78ed85557c8a348832a766639bf44cc70"} Oct 30 21:46:54.801703 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:54.801705 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh" Oct 30 21:46:54.840160 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:54.840100 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh" podStartSLOduration=1.840084313 podStartE2EDuration="1.840084313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:54.838265822 +0000 UTC m=+1083.434082502" watchObservedRunningTime="2026-10-30 21:46:54.840084313 +0000 UTC m=+1083.435900980" Oct 30 21:46:55.120138 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:55.120099 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:55.525542 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:55.525463 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d5acfa7-3679-4ddf-91d4-db8d650f84b3" path="/var/lib/kubelet/pods/7d5acfa7-3679-4ddf-91d4-db8d650f84b3/volumes" Oct 30 21:46:55.835371 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:55.835332 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:55.966782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:55.961542 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:56.055712 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:56.055646 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3003efe5-ppb5f"] Oct 30 21:46:56.085809 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:56.085135 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3003efe5-ppb5f"] Oct 30 21:46:56.735171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:56.735135 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:56.819777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:56.819727 5801 scope.go:122] "RemoveContainer" containerID="f89d8ed1a025e9a37f57c52692c407e21dcd73c1d5e49e4347e6dcd4744e8368" Oct 30 21:46:56.847590 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:56.847568 5801 scope.go:122] "RemoveContainer" containerID="ffd196756aa7d915588f785d05c6a12581c623451450668b89add264129063fa" Oct 30 21:46:56.860978 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:56.860957 5801 scope.go:122] "RemoveContainer" containerID="cfae2044b1b0edd0ab1302891e7532f8abe2f1abca54279b728e4654a99146e3" Oct 30 21:46:56.873804 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:56.873781 5801 scope.go:122] "RemoveContainer" containerID="c88b73d1245a102a6039862596f35bbdbe645a26b2137eac667837cd8ffc4012" Oct 30 21:46:56.885639 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:56.885612 5801 scope.go:122] "RemoveContainer" containerID="bd661adbe59eefe3265d1abd39b46fd24b64df2cddb3653812eb004413d06b7b" Oct 30 21:46:56.896240 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:56.896217 5801 scope.go:122] "RemoveContainer" containerID="4de4fa98443d1099f1e364956f6235796c70a34af09a32cedf703ed4fc63b9da" Oct 30 21:46:57.029083 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:57.029005 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed945f27-nrnqc"] Oct 30 21:46:57.034070 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:57.034046 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed945f27-nrnqc"] Oct 30 21:46:57.450816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:57.450742 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:57.537143 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:57.537102 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a7b59b3-0b71-4359-8918-b0ed7a8d1496" path="/var/lib/kubelet/pods/8a7b59b3-0b71-4359-8918-b0ed7a8d1496/volumes" Oct 30 21:46:57.539001 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:57.538084 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f264c6aa-54c1-41cd-9012-334aef4c85f2" path="/var/lib/kubelet/pods/f264c6aa-54c1-41cd-9012-334aef4c85f2/volumes" Oct 30 21:46:57.880365 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:57.880329 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:58.159364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.159284 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:58.345232 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.345185 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt"] Oct 30 21:46:58.352084 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.352059 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.356307 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.356282 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-08aafeac-openshift-default-dockercfg-qg8xv\"" Oct 30 21:46:58.361101 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.361075 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt"] Oct 30 21:46:58.473633 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.473551 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-envoy\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.473633 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.473591 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-podinfo\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.473924 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.473736 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-data\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.473999 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.473926 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-credential-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.474142 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.474119 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-token\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.474225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.474188 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-workload-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.474330 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.474300 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26eafe2-a22a-47a0-ae23-562931b2cead-istiod-ca-cert\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.474447 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.474346 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kzrd\" (UniqueName: \"kubernetes.io/projected/a26eafe2-a22a-47a0-ae23-562931b2cead-kube-api-access-7kzrd\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.474528 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.474460 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-workload-certs\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.576557 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.576523 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-credential-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.576736 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.576616 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-token\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.576736 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.576649 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-workload-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.576887 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.576789 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26eafe2-a22a-47a0-ae23-562931b2cead-istiod-ca-cert\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.576887 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.576849 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kzrd\" (UniqueName: \"kubernetes.io/projected/a26eafe2-a22a-47a0-ae23-562931b2cead-kube-api-access-7kzrd\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.576963 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.576906 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-workload-certs\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.577068 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.577039 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-envoy\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.577068 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.577057 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-credential-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.577253 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.577075 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-podinfo\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.577253 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.577093 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-workload-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.577253 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.577134 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-data\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.577426 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.577272 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-workload-certs\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.577506 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.577484 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-data\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.577506 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.577498 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26eafe2-a22a-47a0-ae23-562931b2cead-istiod-ca-cert\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.579340 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.579320 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-envoy\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.579436 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.579419 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-podinfo\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.594739 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.594716 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-token\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.597249 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.597224 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kzrd\" (UniqueName: \"kubernetes.io/projected/a26eafe2-a22a-47a0-ae23-562931b2cead-kube-api-access-7kzrd\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.671234 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.671195 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:46:58.969599 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:46:58.969562 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda26eafe2_a22a_47a0_ae23_562931b2cead.slice/crio-a7d0fa460c0c5bdb2e4c5a5895c6e4a535017b1921db4e86e1573f5b25ed2792 WatchSource:0}: Error finding container a7d0fa460c0c5bdb2e4c5a5895c6e4a535017b1921db4e86e1573f5b25ed2792: Status 404 returned error can't find the container with id a7d0fa460c0c5bdb2e4c5a5895c6e4a535017b1921db4e86e1573f5b25ed2792 Oct 30 21:46:58.970309 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.970281 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt"] Oct 30 21:46:58.972657 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.972619 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:58.972913 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.972723 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:58.972913 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:58.972783 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:59.038351 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:59.038323 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:59.868154 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:59.868117 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:59.884630 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:59.884595 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" event={"ID":"a26eafe2-a22a-47a0-ae23-562931b2cead","Type":"ContainerStarted","Data":"84614df4a9236c9a2c63fae4cf0542c00e1e9b6397ceb861b2a3da72f0b70333"} Oct 30 21:46:59.884858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:59.884648 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" event={"ID":"a26eafe2-a22a-47a0-ae23-562931b2cead","Type":"ContainerStarted","Data":"a7d0fa460c0c5bdb2e4c5a5895c6e4a535017b1921db4e86e1573f5b25ed2792"} Oct 30 21:46:59.920637 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:46:59.920564 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" podStartSLOduration=1.920539587 podStartE2EDuration="1.920539587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:59.912322165 +0000 UTC m=+1088.508138852" watchObservedRunningTime="2026-10-30 21:46:59.920539587 +0000 UTC m=+1088.516356249" Oct 30 21:47:00.030099 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:00.030066 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d08887a-ckqgm"] Oct 30 21:47:00.035798 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:00.035765 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d08887a-ckqgm"] Oct 30 21:47:00.304718 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:47:00.304681 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:47:00.673215 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:00.673124 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:47:00.679538 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:00.679503 5801 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 21:47:00.679702 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:00.679558 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" podUID="a26eafe2-a22a-47a0-ae23-562931b2cead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 21:47:00.745604 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:00.745570 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:01.431273 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:01.431235 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:01.531292 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:01.531252 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f2f31c2-d994-43c8-9213-040859e880ff" path="/var/lib/kubelet/pods/3f2f31c2-d994-43c8-9213-040859e880ff/volumes" Oct 30 21:47:01.678778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:01.678650 5801 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 21:47:01.678778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:01.678725 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" podUID="a26eafe2-a22a-47a0-ae23-562931b2cead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 21:47:02.178384 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:02.178335 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:02.672388 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:02.672187 5801 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 21:47:02.673007 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:02.672976 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" podUID="a26eafe2-a22a-47a0-ae23-562931b2cead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 21:47:02.873970 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:02.873933 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:03.545262 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:03.545226 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:03.694778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:03.692137 5801 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 21:47:03.694778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:03.692213 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" podUID="a26eafe2-a22a-47a0-ae23-562931b2cead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 21:47:04.429778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:04.429146 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:04.547533 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:04.547496 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:04.651127 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:04.651093 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:04.671603 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:04.671568 5801 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 21:47:04.672002 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:04.671973 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" podUID="a26eafe2-a22a-47a0-ae23-562931b2cead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 21:47:05.452835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:05.449651 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:05.672774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:05.672232 5801 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 21:47:05.672774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:05.672356 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" podUID="a26eafe2-a22a-47a0-ae23-562931b2cead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 21:47:05.815329 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:05.815299 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh" Oct 30 21:47:06.490338 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:06.490297 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:06.699813 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:06.699076 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:47:06.703077 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:06.700979 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:47:06.711243 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:06.711216 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:47:07.273737 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:07.273695 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:07.979776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:07.979118 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:08.404303 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:08.404268 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:08.642007 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:08.641969 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:08.949524 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:08.949489 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:09.215155 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:09.215065 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:09.681780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:09.681644 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:10.404046 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:10.403963 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:11.199836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.198742 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:11.678775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.676359 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k"] Oct 30 21:47:11.708657 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.708619 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k"] Oct 30 21:47:11.708900 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.708734 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.800607 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.800580 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01540444-bx7tl"] Oct 30 21:47:11.807215 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.807190 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01540444-bx7tl" Oct 30 21:47:11.809717 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.809686 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01540444-bx7tl"] Oct 30 21:47:11.814282 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.814253 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:47:11.814559 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.814543 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:47:11.853614 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.853582 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:47:11.853930 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.853603 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:47:11.860814 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.859329 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-workload-certs\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.860814 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.859412 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpjsr\" (UniqueName: \"kubernetes.io/projected/402d335b-8671-4c9a-957e-be90693d2085-kube-api-access-lpjsr\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.860814 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.859575 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-workload-socket\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.860814 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.859605 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-credential-socket\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.860814 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.859632 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-istio-envoy\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.860814 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.859799 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/402d335b-8671-4c9a-957e-be90693d2085-istiod-ca-cert\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.860814 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.859846 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/402d335b-8671-4c9a-957e-be90693d2085-istio-podinfo\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.860814 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.860142 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/402d335b-8671-4c9a-957e-be90693d2085-istio-token\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.860814 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.860197 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-istio-data\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.966839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.965997 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fb89951-0ce2-4023-a112-7ea5fd56b4a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01540444-bx7tl\" (UID: \"4fb89951-0ce2-4023-a112-7ea5fd56b4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01540444-bx7tl" Oct 30 21:47:11.966839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.966121 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/402d335b-8671-4c9a-957e-be90693d2085-istiod-ca-cert\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.966839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.966167 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/402d335b-8671-4c9a-957e-be90693d2085-istio-podinfo\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.966839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.966311 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/402d335b-8671-4c9a-957e-be90693d2085-istio-token\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.966839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.966352 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsds8\" (UniqueName: \"kubernetes.io/projected/4fb89951-0ce2-4023-a112-7ea5fd56b4a2-kube-api-access-gsds8\") pod \"maas-api-revoke-sub-01540444-bx7tl\" (UID: \"4fb89951-0ce2-4023-a112-7ea5fd56b4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01540444-bx7tl" Oct 30 21:47:11.966839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.966379 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-istio-data\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.966839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.966494 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-workload-certs\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.966839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.966537 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fb89951-0ce2-4023-a112-7ea5fd56b4a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01540444-bx7tl\" (UID: \"4fb89951-0ce2-4023-a112-7ea5fd56b4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01540444-bx7tl" Oct 30 21:47:11.966839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.966592 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpjsr\" (UniqueName: \"kubernetes.io/projected/402d335b-8671-4c9a-957e-be90693d2085-kube-api-access-lpjsr\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.966839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.966746 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-workload-socket\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.966839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.966794 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-credential-socket\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.966839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.966823 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-istio-envoy\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.980352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.980282 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-istio-envoy\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.981462 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.981435 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/402d335b-8671-4c9a-957e-be90693d2085-istiod-ca-cert\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.984166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.984136 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-workload-certs\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.986397 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.984765 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-istio-data\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.986397 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.985007 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/402d335b-8671-4c9a-957e-be90693d2085-istio-podinfo\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.986397 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.985054 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-workload-socket\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:11.986397 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:11.985171 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-credential-socket\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:12.000656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:12.000621 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpjsr\" (UniqueName: \"kubernetes.io/projected/402d335b-8671-4c9a-957e-be90693d2085-kube-api-access-lpjsr\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:12.000817 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:12.000741 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/402d335b-8671-4c9a-957e-be90693d2085-istio-token\") pod \"e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:12.030344 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:12.030046 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:12.092013 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:12.090928 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fb89951-0ce2-4023-a112-7ea5fd56b4a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01540444-bx7tl\" (UID: \"4fb89951-0ce2-4023-a112-7ea5fd56b4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01540444-bx7tl" Oct 30 21:47:12.092013 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:12.091169 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsds8\" (UniqueName: \"kubernetes.io/projected/4fb89951-0ce2-4023-a112-7ea5fd56b4a2-kube-api-access-gsds8\") pod \"maas-api-revoke-sub-01540444-bx7tl\" (UID: \"4fb89951-0ce2-4023-a112-7ea5fd56b4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01540444-bx7tl" Oct 30 21:47:12.092013 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:12.091283 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fb89951-0ce2-4023-a112-7ea5fd56b4a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01540444-bx7tl\" (UID: \"4fb89951-0ce2-4023-a112-7ea5fd56b4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01540444-bx7tl" Oct 30 21:47:12.092327 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:12.092106 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fb89951-0ce2-4023-a112-7ea5fd56b4a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01540444-bx7tl\" (UID: \"4fb89951-0ce2-4023-a112-7ea5fd56b4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01540444-bx7tl" Oct 30 21:47:12.096177 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:12.096145 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fb89951-0ce2-4023-a112-7ea5fd56b4a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01540444-bx7tl\" (UID: \"4fb89951-0ce2-4023-a112-7ea5fd56b4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01540444-bx7tl" Oct 30 21:47:12.105700 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:12.105671 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsds8\" (UniqueName: \"kubernetes.io/projected/4fb89951-0ce2-4023-a112-7ea5fd56b4a2-kube-api-access-gsds8\") pod \"maas-api-revoke-sub-01540444-bx7tl\" (UID: \"4fb89951-0ce2-4023-a112-7ea5fd56b4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01540444-bx7tl" Oct 30 21:47:12.145163 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:12.145129 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01540444-bx7tl" Oct 30 21:47:12.181411 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:12.181327 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:12.403590 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:12.403518 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k"] Oct 30 21:47:12.408318 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:12.408273 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:47:12.408451 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:12.408370 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:47:12.408451 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:12.408411 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:47:12.512538 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:12.512492 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" event={"ID":"402d335b-8671-4c9a-957e-be90693d2085","Type":"ContainerStarted","Data":"69ca82f5bf4ddef2211ed736c343b1d3aa2771c7f002cc17e04311ed032b5540"} Oct 30 21:47:12.679576 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:12.679541 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01540444-bx7tl"] Oct 30 21:47:13.230666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:13.230626 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:13.531812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:13.531723 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01540444-bx7tl" event={"ID":"4fb89951-0ce2-4023-a112-7ea5fd56b4a2","Type":"ContainerStarted","Data":"517c922dba9fc4d4bfe5a76dcadc236cc9d896c9726723b88f77e3035f2180a1"} Oct 30 21:47:13.531812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:13.531788 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01540444-bx7tl" event={"ID":"4fb89951-0ce2-4023-a112-7ea5fd56b4a2","Type":"ContainerStarted","Data":"1225b8b6dd567f0e0e8f833915d3e420d434f30d9cdaf91331b02c27a0c02288"} Oct 30 21:47:13.531812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:13.531811 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" event={"ID":"402d335b-8671-4c9a-957e-be90693d2085","Type":"ContainerStarted","Data":"c0d0345dfbbf1d70d1bc2773bfa220919de7091a2ed8bfec43baac00d734827f"} Oct 30 21:47:13.564843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:13.563396 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01540444-bx7tl" podStartSLOduration=2.56337075 podStartE2EDuration="2.56337075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:13.551610544 +0000 UTC m=+1102.147427219" watchObservedRunningTime="2026-10-30 21:47:13.56337075 +0000 UTC m=+1102.159187426" Oct 30 21:47:13.608934 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:13.608881 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" podStartSLOduration=2.608866756 podStartE2EDuration="2.608866756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:13.608319096 +0000 UTC m=+1102.204135776" watchObservedRunningTime="2026-10-30 21:47:13.608866756 +0000 UTC m=+1102.204683416" Oct 30 21:47:13.689558 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:13.689502 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:13.886596 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:13.886556 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:14.031398 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:14.031362 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:14.033885 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:14.033856 5801 patch_prober.go:29] interesting pod/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 21:47:14.034022 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:14.033916 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" podUID="402d335b-8671-4c9a-957e-be90693d2085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 21:47:14.342453 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:14.342408 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:14.545084 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:14.545046 5801 generic.go:356] "Generic (PLEG): container finished" podID="4fb89951-0ce2-4023-a112-7ea5fd56b4a2" containerID="517c922dba9fc4d4bfe5a76dcadc236cc9d896c9726723b88f77e3035f2180a1" exitCode=0 Oct 30 21:47:14.545408 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:14.545380 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01540444-bx7tl" event={"ID":"4fb89951-0ce2-4023-a112-7ea5fd56b4a2","Type":"ContainerDied","Data":"517c922dba9fc4d4bfe5a76dcadc236cc9d896c9726723b88f77e3035f2180a1"} Oct 30 21:47:14.880778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:14.879810 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:15.004015 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.003945 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:15.085777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.084910 5801 patch_prober.go:29] interesting pod/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 21:47:15.085777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.084978 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" podUID="402d335b-8671-4c9a-957e-be90693d2085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 21:47:15.097919 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.097594 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed2fc544-ln794"] Oct 30 21:47:15.120604 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.114591 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c0755d-fgfdv"] Oct 30 21:47:15.124140 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.124062 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed2fc544-ln794"] Oct 30 21:47:15.126265 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.124320 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:15.130697 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.130663 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c0755d-fgfdv"] Oct 30 21:47:15.132973 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.132903 5801 patch_prober.go:29] interesting pod/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:47:15.132973 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.132959 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" podUID="929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:15.194559 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.194512 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t"] Oct 30 21:47:15.206684 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.202681 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t" Oct 30 21:47:15.211572 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.211536 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t"] Oct 30 21:47:15.217979 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.217842 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-cd3c4c-dockercfg-jhrhb\"" Oct 30 21:47:15.237943 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.237789 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:15.301098 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.301059 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-cd3c4c\"" Oct 30 21:47:15.338996 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.333585 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bd5b368-4306-4836-b13a-e3ada8861d3b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t\" (UID: \"6bd5b368-4306-4836-b13a-e3ada8861d3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t" Oct 30 21:47:15.338996 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.333819 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnffb\" (UniqueName: \"kubernetes.io/projected/6bd5b368-4306-4836-b13a-e3ada8861d3b-kube-api-access-jnffb\") pod \"payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t\" (UID: \"6bd5b368-4306-4836-b13a-e3ada8861d3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t" Oct 30 21:47:15.493836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.493285 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bd5b368-4306-4836-b13a-e3ada8861d3b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t\" (UID: \"6bd5b368-4306-4836-b13a-e3ada8861d3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t" Oct 30 21:47:15.493836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.493512 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnffb\" (UniqueName: \"kubernetes.io/projected/6bd5b368-4306-4836-b13a-e3ada8861d3b-kube-api-access-jnffb\") pod \"payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t\" (UID: \"6bd5b368-4306-4836-b13a-e3ada8861d3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t" Oct 30 21:47:15.507774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.494381 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:15.507774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.494887 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bd5b368-4306-4836-b13a-e3ada8861d3b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t\" (UID: \"6bd5b368-4306-4836-b13a-e3ada8861d3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t" Oct 30 21:47:15.629818 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.624043 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:15.655306 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.640780 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a913f7e1-ae52-414b-a1fe-be53429bcf11" path="/var/lib/kubelet/pods/a913f7e1-ae52-414b-a1fe-be53429bcf11/volumes" Oct 30 21:47:15.655306 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.641577 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b07cdfb9-6011-4083-9743-8d3503646a95" path="/var/lib/kubelet/pods/b07cdfb9-6011-4083-9743-8d3503646a95/volumes" Oct 30 21:47:15.688775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.687561 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnffb\" (UniqueName: \"kubernetes.io/projected/6bd5b368-4306-4836-b13a-e3ada8861d3b-kube-api-access-jnffb\") pod \"payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t\" (UID: \"6bd5b368-4306-4836-b13a-e3ada8861d3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t" Oct 30 21:47:15.737775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.733518 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:15.832082 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.826437 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t" Oct 30 21:47:15.965591 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:15.965556 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:16.031530 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:16.031179 5801 patch_prober.go:29] interesting pod/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 21:47:16.031530 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:16.031236 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" podUID="402d335b-8671-4c9a-957e-be90693d2085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 21:47:16.104839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:16.104361 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01540444-bx7tl" Oct 30 21:47:16.141590 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:16.141466 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fb89951-0ce2-4023-a112-7ea5fd56b4a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fb89951-0ce2-4023-a112-7ea5fd56b4a2-maas-api-openshift-ca\") pod \"4fb89951-0ce2-4023-a112-7ea5fd56b4a2\" (UID: \"4fb89951-0ce2-4023-a112-7ea5fd56b4a2\") " Oct 30 21:47:16.141590 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:16.141583 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fb89951-0ce2-4023-a112-7ea5fd56b4a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fb89951-0ce2-4023-a112-7ea5fd56b4a2-maas-api-xks-ca\") pod \"4fb89951-0ce2-4023-a112-7ea5fd56b4a2\" (UID: \"4fb89951-0ce2-4023-a112-7ea5fd56b4a2\") " Oct 30 21:47:16.142035 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:16.141699 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fb89951-0ce2-4023-a112-7ea5fd56b4a2-kube-api-access-gsds8\" (UniqueName: \"kubernetes.io/projected/4fb89951-0ce2-4023-a112-7ea5fd56b4a2-kube-api-access-gsds8\") pod \"4fb89951-0ce2-4023-a112-7ea5fd56b4a2\" (UID: \"4fb89951-0ce2-4023-a112-7ea5fd56b4a2\") " Oct 30 21:47:16.142792 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:16.142200 5801 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:47:16.142792 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:16.142256 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" podUID="a7e5545a-7bc6-437b-bef6-4c6ee6776f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:16.143158 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:16.143119 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fb89951-0ce2-4023-a112-7ea5fd56b4a2-maas-api-openshift-ca" pod "4fb89951-0ce2-4023-a112-7ea5fd56b4a2" (UID: "4fb89951-0ce2-4023-a112-7ea5fd56b4a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:16.145782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:16.145743 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fb89951-0ce2-4023-a112-7ea5fd56b4a2-maas-api-xks-ca" pod "4fb89951-0ce2-4023-a112-7ea5fd56b4a2" (UID: "4fb89951-0ce2-4023-a112-7ea5fd56b4a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:16.184100 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:16.182979 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb89951-0ce2-4023-a112-7ea5fd56b4a2-kube-api-access-gsds8" pod "4fb89951-0ce2-4023-a112-7ea5fd56b4a2" (UID: "4fb89951-0ce2-4023-a112-7ea5fd56b4a2"). InnerVolumeSpecName "kube-api-access-gsds8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:16.246120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:16.245526 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fb89951-0ce2-4023-a112-7ea5fd56b4a2-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:16.246120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:16.245591 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsds8\" (UniqueName: \"kubernetes.io/projected/4fb89951-0ce2-4023-a112-7ea5fd56b4a2-kube-api-access-gsds8\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:16.246120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:16.245610 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fb89951-0ce2-4023-a112-7ea5fd56b4a2-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:16.386871 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:16.378242 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:16.429193 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:47:16.429077 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd5b368_4306_4836_b13a_e3ada8861d3b.slice/crio-f38b1ed352510ff6f08ffc976167801f54427f445ff2bfe8b9586cbb9f49602f WatchSource:0}: Error finding container f38b1ed352510ff6f08ffc976167801f54427f445ff2bfe8b9586cbb9f49602f: Status 404 returned error can't find the container with id f38b1ed352510ff6f08ffc976167801f54427f445ff2bfe8b9586cbb9f49602f Oct 30 21:47:16.433495 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:16.433456 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t"] Oct 30 21:47:16.681082 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:16.679873 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01540444-bx7tl" event={"ID":"4fb89951-0ce2-4023-a112-7ea5fd56b4a2","Type":"ContainerDied","Data":"1225b8b6dd567f0e0e8f833915d3e420d434f30d9cdaf91331b02c27a0c02288"} Oct 30 21:47:16.681082 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:16.679938 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1225b8b6dd567f0e0e8f833915d3e420d434f30d9cdaf91331b02c27a0c02288" Oct 30 21:47:16.681082 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:16.680035 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01540444-bx7tl" Oct 30 21:47:16.689262 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:16.687834 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t" event={"ID":"6bd5b368-4306-4836-b13a-e3ada8861d3b","Type":"ContainerStarted","Data":"77fa14775c0f12c987c172f86a847c9f20751a56f438dd59a9b64dbebb34e500"} Oct 30 21:47:16.689262 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:16.687897 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t" event={"ID":"6bd5b368-4306-4836-b13a-e3ada8861d3b","Type":"ContainerStarted","Data":"f38b1ed352510ff6f08ffc976167801f54427f445ff2bfe8b9586cbb9f49602f"} Oct 30 21:47:16.689262 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:16.689154 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t" Oct 30 21:47:16.719061 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:16.718990 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t" podStartSLOduration=1.71897309 podStartE2EDuration="1.71897309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:16.717873938 +0000 UTC m=+1105.313690600" watchObservedRunningTime="2026-10-30 21:47:16.71897309 +0000 UTC m=+1105.314789751" Oct 30 21:47:17.031424 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:17.031051 5801 patch_prober.go:29] interesting pod/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 21:47:17.031424 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:17.031098 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" podUID="402d335b-8671-4c9a-957e-be90693d2085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 21:47:17.073519 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:17.073313 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:18.016165 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:18.016128 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:18.031338 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:18.031312 5801 patch_prober.go:29] interesting pod/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 21:47:18.031491 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:18.031389 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" podUID="402d335b-8671-4c9a-957e-be90693d2085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 21:47:18.779366 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:18.778907 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:19.032109 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:19.032024 5801 patch_prober.go:29] interesting pod/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 21:47:19.032109 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:19.032088 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" podUID="402d335b-8671-4c9a-957e-be90693d2085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 21:47:19.545101 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:19.545072 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:19.727159 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:19.727061 5801 generic.go:356] "Generic (PLEG): container finished" podID="a7e5545a-7bc6-437b-bef6-4c6ee6776f09" containerID="d944e9bb4c96c7dcf933f124619b6d599bee4f73f83a6c6712a109b6f60cfb72" exitCode=0 Oct 30 21:47:19.727159 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:19.727143 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" event={"ID":"a7e5545a-7bc6-437b-bef6-4c6ee6776f09","Type":"ContainerDied","Data":"d944e9bb4c96c7dcf933f124619b6d599bee4f73f83a6c6712a109b6f60cfb72"} Oct 30 21:47:19.727680 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:19.727654 5801 scope.go:122] "RemoveContainer" containerID="d944e9bb4c96c7dcf933f124619b6d599bee4f73f83a6c6712a109b6f60cfb72" Oct 30 21:47:19.729273 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:19.729246 5801 generic.go:356] "Generic (PLEG): container finished" podID="929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" containerID="f57ffb3b59ac5ad6024b8b8e95001f7fabea9c4c10c94b3787e929cac75b5b2e" exitCode=0 Oct 30 21:47:19.729382 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:19.729310 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" event={"ID":"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf","Type":"ContainerDied","Data":"f57ffb3b59ac5ad6024b8b8e95001f7fabea9c4c10c94b3787e929cac75b5b2e"} Oct 30 21:47:19.729745 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:19.729727 5801 scope.go:122] "RemoveContainer" containerID="f57ffb3b59ac5ad6024b8b8e95001f7fabea9c4c10c94b3787e929cac75b5b2e" Oct 30 21:47:19.729876 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:19.729812 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:47:19.729939 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:19.729920 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:47:19.729998 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:19.729971 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:47:19.731618 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:19.731587 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:47:19.731731 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:19.731649 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:47:19.731731 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:19.731701 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:47:20.038518 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.038421 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:20.040113 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.040092 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:20.040688 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.040662 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:47:20.168629 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.168598 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:20.380632 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.380602 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m"] Oct 30 21:47:20.381495 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.381474 5801 cpu_manager.go:414] "containerMap: removing container" podUID="4fb89951-0ce2-4023-a112-7ea5fd56b4a2" containerName="revoke-keys" Oct 30 21:47:20.381556 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.381496 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fb89951-0ce2-4023-a112-7ea5fd56b4a2" containerName="revoke-keys" Oct 30 21:47:20.381624 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.381614 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fb89951-0ce2-4023-a112-7ea5fd56b4a2" containerName="revoke-keys" Oct 30 21:47:20.385854 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.385834 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m" Oct 30 21:47:20.389890 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.389865 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:47:20.389890 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.389871 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:47:20.390089 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.390028 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:47:20.390291 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.390268 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:47:20.395929 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.395908 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m"] Oct 30 21:47:20.490184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.490151 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192d484c-6d1c-4d8f-ab93-7f9d011317e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m\" (UID: \"192d484c-6d1c-4d8f-ab93-7f9d011317e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m" Oct 30 21:47:20.490184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.490191 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192d484c-6d1c-4d8f-ab93-7f9d011317e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m\" (UID: \"192d484c-6d1c-4d8f-ab93-7f9d011317e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m" Oct 30 21:47:20.490590 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.490561 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p2s9\" (UniqueName: \"kubernetes.io/projected/192d484c-6d1c-4d8f-ab93-7f9d011317e7-kube-api-access-9p2s9\") pod \"maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m\" (UID: \"192d484c-6d1c-4d8f-ab93-7f9d011317e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m" Oct 30 21:47:20.514772 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.513816 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:20.600780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.599774 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p2s9\" (UniqueName: \"kubernetes.io/projected/192d484c-6d1c-4d8f-ab93-7f9d011317e7-kube-api-access-9p2s9\") pod \"maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m\" (UID: \"192d484c-6d1c-4d8f-ab93-7f9d011317e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m" Oct 30 21:47:20.600780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.600034 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192d484c-6d1c-4d8f-ab93-7f9d011317e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m\" (UID: \"192d484c-6d1c-4d8f-ab93-7f9d011317e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m" Oct 30 21:47:20.600780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.600081 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192d484c-6d1c-4d8f-ab93-7f9d011317e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m\" (UID: \"192d484c-6d1c-4d8f-ab93-7f9d011317e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m" Oct 30 21:47:20.616776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.615569 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192d484c-6d1c-4d8f-ab93-7f9d011317e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m\" (UID: \"192d484c-6d1c-4d8f-ab93-7f9d011317e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m" Oct 30 21:47:20.670714 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.670628 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p2s9\" (UniqueName: \"kubernetes.io/projected/192d484c-6d1c-4d8f-ab93-7f9d011317e7-kube-api-access-9p2s9\") pod \"maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m\" (UID: \"192d484c-6d1c-4d8f-ab93-7f9d011317e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m" Oct 30 21:47:20.671105 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.671082 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192d484c-6d1c-4d8f-ab93-7f9d011317e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m\" (UID: \"192d484c-6d1c-4d8f-ab93-7f9d011317e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m" Oct 30 21:47:20.709843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.707873 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m" Oct 30 21:47:20.750773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.747445 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:20.785341 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.785276 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" event={"ID":"a7e5545a-7bc6-437b-bef6-4c6ee6776f09","Type":"ContainerStarted","Data":"1e072289193d11c9d353f41dabe560444a8737a005c53be22a79c809d6764f16"} Oct 30 21:47:20.790855 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:20.790821 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" event={"ID":"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf","Type":"ContainerStarted","Data":"be4676eb2e66fbcb2ba91df95e765b219028be4ff010ed01d60ba4e8e95f068d"} Oct 30 21:47:21.030294 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:21.030259 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81dc9aa0-4rk86"] Oct 30 21:47:21.035801 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:21.035707 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81dc9aa0-4rk86"] Oct 30 21:47:21.207724 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:47:21.207676 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192d484c_6d1c_4d8f_ab93_7f9d011317e7.slice/crio-7c7c7aa641e5a60118b612d569557847f53d7c55131e79dd3d2cee0db7188f4f WatchSource:0}: Error finding container 7c7c7aa641e5a60118b612d569557847f53d7c55131e79dd3d2cee0db7188f4f: Status 404 returned error can't find the container with id 7c7c7aa641e5a60118b612d569557847f53d7c55131e79dd3d2cee0db7188f4f Oct 30 21:47:21.208491 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:21.208466 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m"] Oct 30 21:47:21.515776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:21.515651 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:21.529640 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:21.529588 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4600e16-207d-45f5-bbb4-55cda56ce625" path="/var/lib/kubelet/pods/c4600e16-207d-45f5-bbb4-55cda56ce625/volumes" Oct 30 21:47:21.745115 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:21.745030 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:47:21.745905 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:21.745410 5801 patch_prober.go:29] interesting pod/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 21:47:21.746084 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:21.746060 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" podUID="929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 21:47:21.782782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:21.780124 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:47:21.782782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:21.780259 5801 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 21:47:21.782782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:21.780303 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" podUID="a7e5545a-7bc6-437b-bef6-4c6ee6776f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 21:47:21.826986 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:21.826950 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m" event={"ID":"192d484c-6d1c-4d8f-ab93-7f9d011317e7","Type":"ContainerStarted","Data":"b13b53269648edfc906c509f64cd08d2fce832b79c53f62c08e15aab522ed55d"} Oct 30 21:47:21.827146 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:21.827014 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m" event={"ID":"192d484c-6d1c-4d8f-ab93-7f9d011317e7","Type":"ContainerStarted","Data":"7c7c7aa641e5a60118b612d569557847f53d7c55131e79dd3d2cee0db7188f4f"} Oct 30 21:47:21.847076 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:21.846280 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m" podStartSLOduration=1.8462634759999998 podStartE2EDuration="1.846263476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:21.8452937 +0000 UTC m=+1110.441110372" watchObservedRunningTime="2026-10-30 21:47:21.846263476 +0000 UTC m=+1110.442080137" Oct 30 21:47:22.126264 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:22.126233 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:22.371774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:22.363233 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:22.757270 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:22.756665 5801 patch_prober.go:29] interesting pod/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 21:47:22.757270 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:22.756745 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" podUID="929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 21:47:22.791654 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:22.791617 5801 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 21:47:22.791977 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:22.791948 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" podUID="a7e5545a-7bc6-437b-bef6-4c6ee6776f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 21:47:22.846621 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:22.846587 5801 generic.go:356] "Generic (PLEG): container finished" podID="192d484c-6d1c-4d8f-ab93-7f9d011317e7" containerID="b13b53269648edfc906c509f64cd08d2fce832b79c53f62c08e15aab522ed55d" exitCode=0 Oct 30 21:47:22.846831 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:22.846715 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m" event={"ID":"192d484c-6d1c-4d8f-ab93-7f9d011317e7","Type":"ContainerDied","Data":"b13b53269648edfc906c509f64cd08d2fce832b79c53f62c08e15aab522ed55d"} Oct 30 21:47:23.337533 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:23.337492 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:23.740019 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:23.739925 5801 patch_prober.go:29] interesting pod/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 21:47:23.740019 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:23.739986 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" podUID="929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 21:47:23.785780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:23.779249 5801 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 21:47:23.785780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:23.779336 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" podUID="a7e5545a-7bc6-437b-bef6-4c6ee6776f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 21:47:24.045635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:24.045569 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:24.049267 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:24.049236 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd3352f7-vkjzp"] Oct 30 21:47:24.109777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:24.106839 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a019e5ae-x55jc"] Oct 30 21:47:24.202797 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:24.179691 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a019e5ae-x55jc"] Oct 30 21:47:24.202797 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:24.195087 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd3352f7-vkjzp"] Oct 30 21:47:24.407006 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:24.406967 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m" Oct 30 21:47:24.516174 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:24.513708 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/192d484c-6d1c-4d8f-ab93-7f9d011317e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192d484c-6d1c-4d8f-ab93-7f9d011317e7-maas-api-xks-ca\") pod \"192d484c-6d1c-4d8f-ab93-7f9d011317e7\" (UID: \"192d484c-6d1c-4d8f-ab93-7f9d011317e7\") " Oct 30 21:47:24.516174 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:24.513878 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/192d484c-6d1c-4d8f-ab93-7f9d011317e7-kube-api-access-9p2s9\" (UniqueName: \"kubernetes.io/projected/192d484c-6d1c-4d8f-ab93-7f9d011317e7-kube-api-access-9p2s9\") pod \"192d484c-6d1c-4d8f-ab93-7f9d011317e7\" (UID: \"192d484c-6d1c-4d8f-ab93-7f9d011317e7\") " Oct 30 21:47:24.516174 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:24.514865 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/192d484c-6d1c-4d8f-ab93-7f9d011317e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192d484c-6d1c-4d8f-ab93-7f9d011317e7-maas-api-openshift-ca\") pod \"192d484c-6d1c-4d8f-ab93-7f9d011317e7\" (UID: \"192d484c-6d1c-4d8f-ab93-7f9d011317e7\") " Oct 30 21:47:24.516174 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:24.516115 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/192d484c-6d1c-4d8f-ab93-7f9d011317e7-maas-api-openshift-ca" pod "192d484c-6d1c-4d8f-ab93-7f9d011317e7" (UID: "192d484c-6d1c-4d8f-ab93-7f9d011317e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:24.516564 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:24.516489 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192d484c-6d1c-4d8f-ab93-7f9d011317e7-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:24.520195 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:24.520168 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192d484c-6d1c-4d8f-ab93-7f9d011317e7-kube-api-access-9p2s9" pod "192d484c-6d1c-4d8f-ab93-7f9d011317e7" (UID: "192d484c-6d1c-4d8f-ab93-7f9d011317e7"). InnerVolumeSpecName "kube-api-access-9p2s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:24.521113 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:24.521093 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/192d484c-6d1c-4d8f-ab93-7f9d011317e7-maas-api-xks-ca" pod "192d484c-6d1c-4d8f-ab93-7f9d011317e7" (UID: "192d484c-6d1c-4d8f-ab93-7f9d011317e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:24.617952 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:24.617897 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p2s9\" (UniqueName: \"kubernetes.io/projected/192d484c-6d1c-4d8f-ab93-7f9d011317e7-kube-api-access-9p2s9\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:24.617952 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:24.617952 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192d484c-6d1c-4d8f-ab93-7f9d011317e7-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:24.740141 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:24.740108 5801 patch_prober.go:29] interesting pod/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 21:47:24.740508 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:24.740168 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" podUID="929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 21:47:24.746429 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:24.746401 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:24.783829 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:24.783580 5801 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 21:47:24.783829 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:24.783638 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" podUID="a7e5545a-7bc6-437b-bef6-4c6ee6776f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 21:47:25.021333 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:25.021298 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m" Oct 30 21:47:25.021333 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:25.021317 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m" event={"ID":"192d484c-6d1c-4d8f-ab93-7f9d011317e7","Type":"ContainerDied","Data":"7c7c7aa641e5a60118b612d569557847f53d7c55131e79dd3d2cee0db7188f4f"} Oct 30 21:47:25.021562 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:25.021404 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c7c7aa641e5a60118b612d569557847f53d7c55131e79dd3d2cee0db7188f4f" Oct 30 21:47:25.527070 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:25.527014 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:25.527870 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:25.527848 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26f7a9d0-422c-45ad-a928-82a851cbbd34" path="/var/lib/kubelet/pods/26f7a9d0-422c-45ad-a928-82a851cbbd34/volumes" Oct 30 21:47:25.528590 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:25.528575 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="545f6262-2b94-4b51-9688-d08feb89cb59" path="/var/lib/kubelet/pods/545f6262-2b94-4b51-9688-d08feb89cb59/volumes" Oct 30 21:47:25.740503 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:25.740468 5801 patch_prober.go:29] interesting pod/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 21:47:25.741042 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:25.740992 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" podUID="929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 21:47:25.780835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:25.780068 5801 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 21:47:25.780835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:25.780128 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" podUID="a7e5545a-7bc6-437b-bef6-4c6ee6776f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 21:47:26.402086 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:26.401992 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:26.740590 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:26.740504 5801 patch_prober.go:29] interesting pod/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 21:47:26.740590 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:26.740566 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" podUID="929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 21:47:26.776452 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:26.776414 5801 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 21:47:26.776640 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:26.776491 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" podUID="a7e5545a-7bc6-437b-bef6-4c6ee6776f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 21:47:27.040033 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:27.039951 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa8e3a54-z56h7"] Oct 30 21:47:27.044963 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:27.044935 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa8e3a54-z56h7"] Oct 30 21:47:27.191120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:27.191082 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:27.482851 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:27.482817 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:27.526475 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:27.526441 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2efd9b31-0b1e-4b5a-8abd-2041520bc18a" path="/var/lib/kubelet/pods/2efd9b31-0b1e-4b5a-8abd-2041520bc18a/volumes" Oct 30 21:47:27.745791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:27.745690 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:47:27.746452 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:27.746426 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:47:27.748032 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:27.748006 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:47:27.780715 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:27.780689 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:47:27.781131 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:27.781104 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:47:27.781829 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:27.781807 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:47:27.936439 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:27.936403 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:28.037635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:28.037557 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:28.633977 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:28.633939 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:28.720484 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:28.720445 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t" Oct 30 21:47:28.765924 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:28.765710 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:29.245051 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:29.245013 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:29.489796 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:29.487116 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:29.612597 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:29.610404 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:29.714143 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:29.714113 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:29.879110 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:29.878960 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:30.288952 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:47:30.288861 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:47:30.661865 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:30.661627 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:30.758291 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:30.758256 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:31.623201 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:31.623166 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:32.341106 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:32.341065 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:33.038788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:33.038719 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:34.033081 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:34.033044 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:34.673050 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:34.673011 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:34.780213 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:34.780181 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:35.032734 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:35.032653 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfcbbd3b-rh46j"] Oct 30 21:47:35.034556 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:35.034524 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:35.037785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:35.037739 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfcbbd3b-rh46j"] Oct 30 21:47:35.526523 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:35.526484 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0015a27a-eba7-4ed4-8159-4dad866c5211" path="/var/lib/kubelet/pods/0015a27a-eba7-4ed4-8159-4dad866c5211/volumes" Oct 30 21:47:35.699853 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:35.699821 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:35.825480 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:35.825439 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:35.940813 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:35.940741 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:36.292628 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.292539 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:36.338465 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.338435 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j"] Oct 30 21:47:36.339274 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.339256 5801 cpu_manager.go:414] "containerMap: removing container" podUID="192d484c-6d1c-4d8f-ab93-7f9d011317e7" containerName="revoke-keys" Oct 30 21:47:36.339274 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.339275 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="192d484c-6d1c-4d8f-ab93-7f9d011317e7" containerName="revoke-keys" Oct 30 21:47:36.339430 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.339421 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="192d484c-6d1c-4d8f-ab93-7f9d011317e7" containerName="revoke-keys" Oct 30 21:47:36.344380 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.344359 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j" Oct 30 21:47:36.350790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.350765 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-dd5bf8-dockercfg-xfjcl\"" Oct 30 21:47:36.352283 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.352254 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-dd5bf8\"" Oct 30 21:47:36.354446 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.354423 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j"] Oct 30 21:47:36.371679 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.371650 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd"] Oct 30 21:47:36.385062 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.385031 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd" Oct 30 21:47:36.389714 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.389686 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd"] Oct 30 21:47:36.394440 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.394410 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:36.500080 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.500041 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbn2q\" (UniqueName: \"kubernetes.io/projected/8096ecd8-04a2-493d-b0e4-66997cdd6d6f-kube-api-access-gbn2q\") pod \"payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j\" (UID: \"8096ecd8-04a2-493d-b0e4-66997cdd6d6f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j" Oct 30 21:47:36.500364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.500346 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8096ecd8-04a2-493d-b0e4-66997cdd6d6f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j\" (UID: \"8096ecd8-04a2-493d-b0e4-66997cdd6d6f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j" Oct 30 21:47:36.500482 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.500467 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de103d6d-3060-401a-9d10-990b38101c62-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd\" (UID: \"de103d6d-3060-401a-9d10-990b38101c62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd" Oct 30 21:47:36.500634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.500620 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k99hp\" (UniqueName: \"kubernetes.io/projected/de103d6d-3060-401a-9d10-990b38101c62-kube-api-access-k99hp\") pod \"payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd\" (UID: \"de103d6d-3060-401a-9d10-990b38101c62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd" Oct 30 21:47:36.615778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.604324 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbn2q\" (UniqueName: \"kubernetes.io/projected/8096ecd8-04a2-493d-b0e4-66997cdd6d6f-kube-api-access-gbn2q\") pod \"payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j\" (UID: \"8096ecd8-04a2-493d-b0e4-66997cdd6d6f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j" Oct 30 21:47:36.620931 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.620897 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8096ecd8-04a2-493d-b0e4-66997cdd6d6f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j\" (UID: \"8096ecd8-04a2-493d-b0e4-66997cdd6d6f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j" Oct 30 21:47:36.624862 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.624823 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de103d6d-3060-401a-9d10-990b38101c62-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd\" (UID: \"de103d6d-3060-401a-9d10-990b38101c62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd" Oct 30 21:47:36.625023 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.624964 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k99hp\" (UniqueName: \"kubernetes.io/projected/de103d6d-3060-401a-9d10-990b38101c62-kube-api-access-k99hp\") pod \"payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd\" (UID: \"de103d6d-3060-401a-9d10-990b38101c62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd" Oct 30 21:47:36.629776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.626310 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de103d6d-3060-401a-9d10-990b38101c62-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd\" (UID: \"de103d6d-3060-401a-9d10-990b38101c62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd" Oct 30 21:47:36.630872 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.630847 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8096ecd8-04a2-493d-b0e4-66997cdd6d6f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j\" (UID: \"8096ecd8-04a2-493d-b0e4-66997cdd6d6f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j" Oct 30 21:47:36.670773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.665451 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbn2q\" (UniqueName: \"kubernetes.io/projected/8096ecd8-04a2-493d-b0e4-66997cdd6d6f-kube-api-access-gbn2q\") pod \"payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j\" (UID: \"8096ecd8-04a2-493d-b0e4-66997cdd6d6f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j" Oct 30 21:47:36.670773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.667389 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k99hp\" (UniqueName: \"kubernetes.io/projected/de103d6d-3060-401a-9d10-990b38101c62-kube-api-access-k99hp\") pod \"payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd\" (UID: \"de103d6d-3060-401a-9d10-990b38101c62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd" Oct 30 21:47:36.710802 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.710771 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd" Oct 30 21:47:36.963972 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:36.963880 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j" Oct 30 21:47:37.013330 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:37.013278 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:37.092831 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:37.089920 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd"] Oct 30 21:47:37.174262 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:37.174229 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd" event={"ID":"de103d6d-3060-401a-9d10-990b38101c62","Type":"ContainerStarted","Data":"9213a4598135309855dad01221ad733bad7544a5d4a82fee92677dbfa714227b"} Oct 30 21:47:37.276481 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:47:37.276455 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8096ecd8_04a2_493d_b0e4_66997cdd6d6f.slice/crio-46a8fbccc526021a9ac1995ffeb084d10da33fb9ef46b5d293582ab584b45344 WatchSource:0}: Error finding container 46a8fbccc526021a9ac1995ffeb084d10da33fb9ef46b5d293582ab584b45344: Status 404 returned error can't find the container with id 46a8fbccc526021a9ac1995ffeb084d10da33fb9ef46b5d293582ab584b45344 Oct 30 21:47:37.277795 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:37.277768 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j"] Oct 30 21:47:37.844262 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:37.844225 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:38.185635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:38.185541 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j" event={"ID":"8096ecd8-04a2-493d-b0e4-66997cdd6d6f","Type":"ContainerStarted","Data":"5de17218b2baf1488d1fd8c529e2fdb0efc01fdca1f6329f99583b238f3aee5e"} Oct 30 21:47:38.185635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:38.185603 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j" event={"ID":"8096ecd8-04a2-493d-b0e4-66997cdd6d6f","Type":"ContainerStarted","Data":"46a8fbccc526021a9ac1995ffeb084d10da33fb9ef46b5d293582ab584b45344"} Oct 30 21:47:38.185924 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:38.185637 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j" Oct 30 21:47:38.187343 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:38.187314 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd" event={"ID":"de103d6d-3060-401a-9d10-990b38101c62","Type":"ContainerStarted","Data":"caf53e8a9d60618c035abe46943ad87e4775a7f6389997ae64837e3540ce6aba"} Oct 30 21:47:38.187472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:38.187366 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd" Oct 30 21:47:38.224155 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:38.224116 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd" podStartSLOduration=2.22410505 podStartE2EDuration="2.22410505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:38.222430792 +0000 UTC m=+1126.818247451" watchObservedRunningTime="2026-10-30 21:47:38.22410505 +0000 UTC m=+1126.819921704" Oct 30 21:47:38.224351 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:38.224334 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j" podStartSLOduration=2.224328362 podStartE2EDuration="2.224328362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:38.20481512 +0000 UTC m=+1126.800631782" watchObservedRunningTime="2026-10-30 21:47:38.224328362 +0000 UTC m=+1126.820145023" Oct 30 21:47:38.670345 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:38.670307 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:39.444346 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:39.444298 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:40.157937 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:40.157898 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:40.344408 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:40.344368 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj"] Oct 30 21:47:40.345056 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:40.345001 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" podUID="7bdabb86-13f1-4ab6-8b08-418fff19b933" containerName="istio-proxy" containerID="cri-o://9caf6898b09fe61c8823dc54a7712a266992f32063b91d42f5d6d8bdeab947f6" gracePeriod=30 Oct 30 21:47:40.354013 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:40.353975 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr"] Oct 30 21:47:40.354425 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:40.354327 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" podUID="24b939ed-32e5-407e-a919-c0395d914dcd" containerName="istio-proxy" containerID="cri-o://e63e393f18cc121add112014195a2b67d7f6c874d342e7b86314fe7bbbdf0237" gracePeriod=30 Oct 30 21:47:40.364698 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:40.364639 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg"] Oct 30 21:47:40.364940 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:40.364887 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" podUID="311939a8-e35d-4636-b052-a368f0a41a9a" containerName="istio-proxy" containerID="cri-o://816b1d8caa5e9ecaf74350b88519d18975c33a8c70a7f4fd8bbfe3c2d182cbec" gracePeriod=30 Oct 30 21:47:41.051173 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:41.051133 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:41.726895 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:41.726860 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:42.469221 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:42.469185 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:42.635560 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:42.635524 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:43.037302 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:43.037271 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-911341b8-dx5th"] Oct 30 21:47:43.042601 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:43.042576 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f722129-qhn4n"] Oct 30 21:47:43.046832 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:43.046816 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-911341b8-dx5th"] Oct 30 21:47:43.051093 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:43.051072 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f722129-qhn4n"] Oct 30 21:47:43.261907 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:43.261869 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:43.531411 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:43.529429 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="675ee6d3-1a83-4ab5-a258-d8445819f14b" path="/var/lib/kubelet/pods/675ee6d3-1a83-4ab5-a258-d8445819f14b/volumes" Oct 30 21:47:43.531411 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:43.530852 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da8e5abc-c29b-49ac-b028-b29521dd8a94" path="/var/lib/kubelet/pods/da8e5abc-c29b-49ac-b028-b29521dd8a94/volumes" Oct 30 21:47:44.032406 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:44.032367 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:44.819716 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:44.819674 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:45.561873 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.561842 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:45.742029 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.741543 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:47:45.768679 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.768629 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:47:45.858037 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.858005 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-token\" (UniqueName: \"kubernetes.io/projected/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-token\") pod \"7bdabb86-13f1-4ab6-8b08-418fff19b933\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " Oct 30 21:47:45.858630 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.858109 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-credential-socket\") pod \"311939a8-e35d-4636-b052-a368f0a41a9a\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " Oct 30 21:47:45.858630 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.858153 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-workload-certs\") pod \"311939a8-e35d-4636-b052-a368f0a41a9a\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " Oct 30 21:47:45.858630 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.858199 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-podinfo\") pod \"7bdabb86-13f1-4ab6-8b08-418fff19b933\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " Oct 30 21:47:45.858630 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.858225 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/311939a8-e35d-4636-b052-a368f0a41a9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311939a8-e35d-4636-b052-a368f0a41a9a-istio-podinfo\") pod \"311939a8-e35d-4636-b052-a368f0a41a9a\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " Oct 30 21:47:45.858630 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.858261 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/311939a8-e35d-4636-b052-a368f0a41a9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311939a8-e35d-4636-b052-a368f0a41a9a-istiod-ca-cert\") pod \"311939a8-e35d-4636-b052-a368f0a41a9a\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " Oct 30 21:47:45.858630 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.858398 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-credential-socket" pod "311939a8-e35d-4636-b052-a368f0a41a9a" (UID: "311939a8-e35d-4636-b052-a368f0a41a9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:45.858630 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.858405 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311939a8-e35d-4636-b052-a368f0a41a9a-kube-api-access-mpjwd\" (UniqueName: \"kubernetes.io/projected/311939a8-e35d-4636-b052-a368f0a41a9a-kube-api-access-mpjwd\") pod \"311939a8-e35d-4636-b052-a368f0a41a9a\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " Oct 30 21:47:45.858630 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.858490 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bdabb86-13f1-4ab6-8b08-418fff19b933-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bdabb86-13f1-4ab6-8b08-418fff19b933-istiod-ca-cert\") pod \"7bdabb86-13f1-4ab6-8b08-418fff19b933\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " Oct 30 21:47:45.859511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.858681 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-istio-data\") pod \"311939a8-e35d-4636-b052-a368f0a41a9a\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " Oct 30 21:47:45.859511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.858804 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-workload-socket\") pod \"7bdabb86-13f1-4ab6-8b08-418fff19b933\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " Oct 30 21:47:45.859511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.858843 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bdabb86-13f1-4ab6-8b08-418fff19b933-kube-api-access-lvjxf\" (UniqueName: \"kubernetes.io/projected/7bdabb86-13f1-4ab6-8b08-418fff19b933-kube-api-access-lvjxf\") pod \"7bdabb86-13f1-4ab6-8b08-418fff19b933\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " Oct 30 21:47:45.859511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.858862 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-workload-certs" pod "311939a8-e35d-4636-b052-a368f0a41a9a" (UID: "311939a8-e35d-4636-b052-a368f0a41a9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:45.859511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.858917 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-workload-socket\") pod \"311939a8-e35d-4636-b052-a368f0a41a9a\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " Oct 30 21:47:45.859511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.858956 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-istio-envoy\") pod \"311939a8-e35d-4636-b052-a368f0a41a9a\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " Oct 30 21:47:45.859511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.859044 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-data\") pod \"7bdabb86-13f1-4ab6-8b08-418fff19b933\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " Oct 30 21:47:45.859511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.859079 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311939a8-e35d-4636-b052-a368f0a41a9a-istio-token\" (UniqueName: \"kubernetes.io/projected/311939a8-e35d-4636-b052-a368f0a41a9a-istio-token\") pod \"311939a8-e35d-4636-b052-a368f0a41a9a\" (UID: \"311939a8-e35d-4636-b052-a368f0a41a9a\") " Oct 30 21:47:45.859511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.859144 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-istio-data" pod "311939a8-e35d-4636-b052-a368f0a41a9a" (UID: "311939a8-e35d-4636-b052-a368f0a41a9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:45.859511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.859149 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-envoy\") pod \"7bdabb86-13f1-4ab6-8b08-418fff19b933\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " Oct 30 21:47:45.859511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.859224 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-credential-socket\") pod \"7bdabb86-13f1-4ab6-8b08-418fff19b933\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " Oct 30 21:47:45.859511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.859287 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-workload-certs\") pod \"7bdabb86-13f1-4ab6-8b08-418fff19b933\" (UID: \"7bdabb86-13f1-4ab6-8b08-418fff19b933\") " Oct 30 21:47:45.860444 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.860417 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:45.860558 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.860447 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:45.860558 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.860462 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:45.860667 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.860641 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-workload-certs" pod "7bdabb86-13f1-4ab6-8b08-418fff19b933" (UID: "7bdabb86-13f1-4ab6-8b08-418fff19b933"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:45.860667 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.860650 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-workload-socket" pod "311939a8-e35d-4636-b052-a368f0a41a9a" (UID: "311939a8-e35d-4636-b052-a368f0a41a9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:45.861036 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.860862 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-workload-socket" pod "7bdabb86-13f1-4ab6-8b08-418fff19b933" (UID: "7bdabb86-13f1-4ab6-8b08-418fff19b933"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:45.861036 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.860878 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-credential-socket" pod "7bdabb86-13f1-4ab6-8b08-418fff19b933" (UID: "7bdabb86-13f1-4ab6-8b08-418fff19b933"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:45.861506 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.861240 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-data" pod "7bdabb86-13f1-4ab6-8b08-418fff19b933" (UID: "7bdabb86-13f1-4ab6-8b08-418fff19b933"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:45.861506 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.861384 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bdabb86-13f1-4ab6-8b08-418fff19b933-istiod-ca-cert" pod "7bdabb86-13f1-4ab6-8b08-418fff19b933" (UID: "7bdabb86-13f1-4ab6-8b08-418fff19b933"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:45.861875 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.861850 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311939a8-e35d-4636-b052-a368f0a41a9a-istiod-ca-cert" pod "311939a8-e35d-4636-b052-a368f0a41a9a" (UID: "311939a8-e35d-4636-b052-a368f0a41a9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:45.862001 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.861979 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-token" pod "7bdabb86-13f1-4ab6-8b08-418fff19b933" (UID: "7bdabb86-13f1-4ab6-8b08-418fff19b933"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:45.862453 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.862428 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311939a8-e35d-4636-b052-a368f0a41a9a-kube-api-access-mpjwd" pod "311939a8-e35d-4636-b052-a368f0a41a9a" (UID: "311939a8-e35d-4636-b052-a368f0a41a9a"). InnerVolumeSpecName "kube-api-access-mpjwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:45.862638 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.862613 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-envoy" pod "7bdabb86-13f1-4ab6-8b08-418fff19b933" (UID: "7bdabb86-13f1-4ab6-8b08-418fff19b933"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:45.862761 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.862722 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-podinfo" pod "7bdabb86-13f1-4ab6-8b08-418fff19b933" (UID: "7bdabb86-13f1-4ab6-8b08-418fff19b933"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:47:45.863354 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.863319 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311939a8-e35d-4636-b052-a368f0a41a9a-istio-token" pod "311939a8-e35d-4636-b052-a368f0a41a9a" (UID: "311939a8-e35d-4636-b052-a368f0a41a9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:45.864258 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.864230 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/311939a8-e35d-4636-b052-a368f0a41a9a-istio-podinfo" pod "311939a8-e35d-4636-b052-a368f0a41a9a" (UID: "311939a8-e35d-4636-b052-a368f0a41a9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:47:45.864338 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.864239 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-istio-envoy" pod "311939a8-e35d-4636-b052-a368f0a41a9a" (UID: "311939a8-e35d-4636-b052-a368f0a41a9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:45.864399 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.864381 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bdabb86-13f1-4ab6-8b08-418fff19b933-kube-api-access-lvjxf" pod "7bdabb86-13f1-4ab6-8b08-418fff19b933" (UID: "7bdabb86-13f1-4ab6-8b08-418fff19b933"). InnerVolumeSpecName "kube-api-access-lvjxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:45.963014 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.962977 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:45.963172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.963017 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvjxf\" (UniqueName: \"kubernetes.io/projected/7bdabb86-13f1-4ab6-8b08-418fff19b933-kube-api-access-lvjxf\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:45.963172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.963034 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:45.963172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.963048 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311939a8-e35d-4636-b052-a368f0a41a9a-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:45.963172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.963063 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:45.963172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.963077 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311939a8-e35d-4636-b052-a368f0a41a9a-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:45.963172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.963129 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:45.963172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.963143 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:45.963172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.963156 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bdabb86-13f1-4ab6-8b08-418fff19b933-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:45.963172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.963170 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:45.963654 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.963184 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bdabb86-13f1-4ab6-8b08-418fff19b933-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:45.963654 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.963198 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311939a8-e35d-4636-b052-a368f0a41a9a-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:45.963654 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.963210 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311939a8-e35d-4636-b052-a368f0a41a9a-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:45.963654 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.963223 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpjwd\" (UniqueName: \"kubernetes.io/projected/311939a8-e35d-4636-b052-a368f0a41a9a-kube-api-access-mpjwd\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:45.963654 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:45.963236 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bdabb86-13f1-4ab6-8b08-418fff19b933-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:46.032167 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.032143 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:47:46.165948 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.165913 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-credential-socket\") pod \"24b939ed-32e5-407e-a919-c0395d914dcd\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " Oct 30 21:47:46.166095 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.166066 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b939ed-32e5-407e-a919-c0395d914dcd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b939ed-32e5-407e-a919-c0395d914dcd-istiod-ca-cert\") pod \"24b939ed-32e5-407e-a919-c0395d914dcd\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " Oct 30 21:47:46.166095 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.166090 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-istio-envoy\") pod \"24b939ed-32e5-407e-a919-c0395d914dcd\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " Oct 30 21:47:46.166219 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.166118 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-workload-certs\") pod \"24b939ed-32e5-407e-a919-c0395d914dcd\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " Oct 30 21:47:46.166219 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.166160 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-credential-socket" pod "24b939ed-32e5-407e-a919-c0395d914dcd" (UID: "24b939ed-32e5-407e-a919-c0395d914dcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:46.166340 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.166234 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-istio-data\") pod \"24b939ed-32e5-407e-a919-c0395d914dcd\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " Oct 30 21:47:46.166340 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.166258 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24b939ed-32e5-407e-a919-c0395d914dcd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b939ed-32e5-407e-a919-c0395d914dcd-istio-podinfo\") pod \"24b939ed-32e5-407e-a919-c0395d914dcd\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " Oct 30 21:47:46.166340 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.166295 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b939ed-32e5-407e-a919-c0395d914dcd-kube-api-access-fgwc4\" (UniqueName: \"kubernetes.io/projected/24b939ed-32e5-407e-a919-c0395d914dcd-kube-api-access-fgwc4\") pod \"24b939ed-32e5-407e-a919-c0395d914dcd\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " Oct 30 21:47:46.166498 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.166350 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-workload-socket\") pod \"24b939ed-32e5-407e-a919-c0395d914dcd\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " Oct 30 21:47:46.166498 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.166400 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-workload-certs" pod "24b939ed-32e5-407e-a919-c0395d914dcd" (UID: "24b939ed-32e5-407e-a919-c0395d914dcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:46.166615 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.166514 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b939ed-32e5-407e-a919-c0395d914dcd-istio-token\" (UniqueName: \"kubernetes.io/projected/24b939ed-32e5-407e-a919-c0395d914dcd-istio-token\") pod \"24b939ed-32e5-407e-a919-c0395d914dcd\" (UID: \"24b939ed-32e5-407e-a919-c0395d914dcd\") " Oct 30 21:47:46.166615 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.166582 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-istio-data" pod "24b939ed-32e5-407e-a919-c0395d914dcd" (UID: "24b939ed-32e5-407e-a919-c0395d914dcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:46.166615 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.166586 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b939ed-32e5-407e-a919-c0395d914dcd-istiod-ca-cert" pod "24b939ed-32e5-407e-a919-c0395d914dcd" (UID: "24b939ed-32e5-407e-a919-c0395d914dcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:46.166806 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.166692 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-workload-socket" pod "24b939ed-32e5-407e-a919-c0395d914dcd" (UID: "24b939ed-32e5-407e-a919-c0395d914dcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:46.167297 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.167274 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b939ed-32e5-407e-a919-c0395d914dcd-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:46.167297 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.167295 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:46.167495 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.167304 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:46.167495 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.167317 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:46.167495 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.167331 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:46.168673 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.168651 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b939ed-32e5-407e-a919-c0395d914dcd-istio-token" pod "24b939ed-32e5-407e-a919-c0395d914dcd" (UID: "24b939ed-32e5-407e-a919-c0395d914dcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:46.168787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.168679 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b939ed-32e5-407e-a919-c0395d914dcd-kube-api-access-fgwc4" pod "24b939ed-32e5-407e-a919-c0395d914dcd" (UID: "24b939ed-32e5-407e-a919-c0395d914dcd"). InnerVolumeSpecName "kube-api-access-fgwc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:46.168883 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.168855 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24b939ed-32e5-407e-a919-c0395d914dcd-istio-podinfo" pod "24b939ed-32e5-407e-a919-c0395d914dcd" (UID: "24b939ed-32e5-407e-a919-c0395d914dcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:47:46.168883 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.168871 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-istio-envoy" pod "24b939ed-32e5-407e-a919-c0395d914dcd" (UID: "24b939ed-32e5-407e-a919-c0395d914dcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:46.199328 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.199266 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-833acf51-m7587"] Oct 30 21:47:46.200040 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.200023 5801 cpu_manager.go:414] "containerMap: removing container" podUID="7bdabb86-13f1-4ab6-8b08-418fff19b933" containerName="istio-proxy" Oct 30 21:47:46.200089 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.200041 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bdabb86-13f1-4ab6-8b08-418fff19b933" containerName="istio-proxy" Oct 30 21:47:46.200089 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.200053 5801 cpu_manager.go:414] "containerMap: removing container" podUID="24b939ed-32e5-407e-a919-c0395d914dcd" containerName="istio-proxy" Oct 30 21:47:46.200089 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.200059 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b939ed-32e5-407e-a919-c0395d914dcd" containerName="istio-proxy" Oct 30 21:47:46.200089 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.200070 5801 cpu_manager.go:414] "containerMap: removing container" podUID="311939a8-e35d-4636-b052-a368f0a41a9a" containerName="istio-proxy" Oct 30 21:47:46.200089 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.200075 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311939a8-e35d-4636-b052-a368f0a41a9a" containerName="istio-proxy" Oct 30 21:47:46.200288 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.200201 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bdabb86-13f1-4ab6-8b08-418fff19b933" containerName="istio-proxy" Oct 30 21:47:46.200288 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.200212 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b939ed-32e5-407e-a919-c0395d914dcd" containerName="istio-proxy" Oct 30 21:47:46.200288 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.200220 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="311939a8-e35d-4636-b052-a368f0a41a9a" containerName="istio-proxy" Oct 30 21:47:46.204171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.204155 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-833acf51-m7587" Oct 30 21:47:46.211722 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.211683 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:47:46.211841 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.211813 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:47:46.211841 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.211827 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:47:46.214045 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.214027 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:47:46.221465 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.221444 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-833acf51-m7587"] Oct 30 21:47:46.269204 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.269176 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b939ed-32e5-407e-a919-c0395d914dcd-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:46.269204 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.269202 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgwc4\" (UniqueName: \"kubernetes.io/projected/24b939ed-32e5-407e-a919-c0395d914dcd-kube-api-access-fgwc4\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:46.269366 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.269213 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b939ed-32e5-407e-a919-c0395d914dcd-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:46.269366 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.269222 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b939ed-32e5-407e-a919-c0395d914dcd-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:46.412965 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.412927 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfzgq\" (UniqueName: \"kubernetes.io/projected/a471144f-13fa-42dc-870a-7022db45e7db-kube-api-access-xfzgq\") pod \"maas-api-revoke-sub-833acf51-m7587\" (UID: \"a471144f-13fa-42dc-870a-7022db45e7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-833acf51-m7587" Oct 30 21:47:46.413191 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.412982 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a471144f-13fa-42dc-870a-7022db45e7db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-833acf51-m7587\" (UID: \"a471144f-13fa-42dc-870a-7022db45e7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-833acf51-m7587" Oct 30 21:47:46.414727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.413475 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a471144f-13fa-42dc-870a-7022db45e7db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-833acf51-m7587\" (UID: \"a471144f-13fa-42dc-870a-7022db45e7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-833acf51-m7587" Oct 30 21:47:46.440799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.437154 5801 generic.go:356] "Generic (PLEG): container finished" podID="7bdabb86-13f1-4ab6-8b08-418fff19b933" containerID="9caf6898b09fe61c8823dc54a7712a266992f32063b91d42f5d6d8bdeab947f6" exitCode=0 Oct 30 21:47:46.440799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.437349 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" event={"ID":"7bdabb86-13f1-4ab6-8b08-418fff19b933","Type":"ContainerDied","Data":"9caf6898b09fe61c8823dc54a7712a266992f32063b91d42f5d6d8bdeab947f6"} Oct 30 21:47:46.440799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.437398 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" event={"ID":"7bdabb86-13f1-4ab6-8b08-418fff19b933","Type":"ContainerDied","Data":"76d450422d5595f476b29903f3567120a24364eab18fa66ed5e5b6f4e020620e"} Oct 30 21:47:46.440799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.437423 5801 scope.go:122] "RemoveContainer" containerID="9caf6898b09fe61c8823dc54a7712a266992f32063b91d42f5d6d8bdeab947f6" Oct 30 21:47:46.440799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.437574 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj" Oct 30 21:47:46.517278 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.509145 5801 generic.go:356] "Generic (PLEG): container finished" podID="24b939ed-32e5-407e-a919-c0395d914dcd" containerID="e63e393f18cc121add112014195a2b67d7f6c874d342e7b86314fe7bbbdf0237" exitCode=0 Oct 30 21:47:46.517278 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.509419 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" event={"ID":"24b939ed-32e5-407e-a919-c0395d914dcd","Type":"ContainerDied","Data":"e63e393f18cc121add112014195a2b67d7f6c874d342e7b86314fe7bbbdf0237"} Oct 30 21:47:46.517278 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.509487 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" event={"ID":"24b939ed-32e5-407e-a919-c0395d914dcd","Type":"ContainerDied","Data":"b066fe451c230f8d3bf2ca3ea17da60c1ba926ac67cd720295e56f26d9cdbd08"} Oct 30 21:47:46.517278 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.509596 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr" Oct 30 21:47:46.521924 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.521894 5801 generic.go:356] "Generic (PLEG): container finished" podID="311939a8-e35d-4636-b052-a368f0a41a9a" containerID="816b1d8caa5e9ecaf74350b88519d18975c33a8c70a7f4fd8bbfe3c2d182cbec" exitCode=0 Oct 30 21:47:46.522228 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.522209 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" event={"ID":"311939a8-e35d-4636-b052-a368f0a41a9a","Type":"ContainerDied","Data":"816b1d8caa5e9ecaf74350b88519d18975c33a8c70a7f4fd8bbfe3c2d182cbec"} Oct 30 21:47:46.522376 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.522362 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" event={"ID":"311939a8-e35d-4636-b052-a368f0a41a9a","Type":"ContainerDied","Data":"0c9bc0c22bb9758354bff101d39a35aaa3c61c064673aa083536f00ea3edf0c1"} Oct 30 21:47:46.522549 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.522536 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg" Oct 30 21:47:46.534256 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.527223 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfzgq\" (UniqueName: \"kubernetes.io/projected/a471144f-13fa-42dc-870a-7022db45e7db-kube-api-access-xfzgq\") pod \"maas-api-revoke-sub-833acf51-m7587\" (UID: \"a471144f-13fa-42dc-870a-7022db45e7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-833acf51-m7587" Oct 30 21:47:46.534256 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.527261 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a471144f-13fa-42dc-870a-7022db45e7db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-833acf51-m7587\" (UID: \"a471144f-13fa-42dc-870a-7022db45e7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-833acf51-m7587" Oct 30 21:47:46.534256 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.527497 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a471144f-13fa-42dc-870a-7022db45e7db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-833acf51-m7587\" (UID: \"a471144f-13fa-42dc-870a-7022db45e7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-833acf51-m7587" Oct 30 21:47:46.534256 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.529032 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a471144f-13fa-42dc-870a-7022db45e7db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-833acf51-m7587\" (UID: \"a471144f-13fa-42dc-870a-7022db45e7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-833acf51-m7587" Oct 30 21:47:46.546879 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.546835 5801 scope.go:122] "RemoveContainer" containerID="9caf6898b09fe61c8823dc54a7712a266992f32063b91d42f5d6d8bdeab947f6" Oct 30 21:47:46.553210 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:47:46.553182 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9caf6898b09fe61c8823dc54a7712a266992f32063b91d42f5d6d8bdeab947f6\": container with ID starting with 9caf6898b09fe61c8823dc54a7712a266992f32063b91d42f5d6d8bdeab947f6 not found: ID does not exist" containerID="9caf6898b09fe61c8823dc54a7712a266992f32063b91d42f5d6d8bdeab947f6" Oct 30 21:47:46.554047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.554015 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9caf6898b09fe61c8823dc54a7712a266992f32063b91d42f5d6d8bdeab947f6"} err="failed to get container status \"9caf6898b09fe61c8823dc54a7712a266992f32063b91d42f5d6d8bdeab947f6\": rpc error: code = NotFound desc = could not find container \"9caf6898b09fe61c8823dc54a7712a266992f32063b91d42f5d6d8bdeab947f6\": container with ID starting with 9caf6898b09fe61c8823dc54a7712a266992f32063b91d42f5d6d8bdeab947f6 not found: ID does not exist" Oct 30 21:47:46.554314 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.554300 5801 scope.go:122] "RemoveContainer" containerID="e63e393f18cc121add112014195a2b67d7f6c874d342e7b86314fe7bbbdf0237" Oct 30 21:47:46.557199 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.557160 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a471144f-13fa-42dc-870a-7022db45e7db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-833acf51-m7587\" (UID: \"a471144f-13fa-42dc-870a-7022db45e7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-833acf51-m7587" Oct 30 21:47:46.574151 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.573448 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj"] Oct 30 21:47:46.582025 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.581160 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfzgq\" (UniqueName: \"kubernetes.io/projected/a471144f-13fa-42dc-870a-7022db45e7db-kube-api-access-xfzgq\") pod \"maas-api-revoke-sub-833acf51-m7587\" (UID: \"a471144f-13fa-42dc-870a-7022db45e7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-833acf51-m7587" Oct 30 21:47:46.592859 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.592828 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-vcdcj"] Oct 30 21:47:46.659431 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.601571 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr"] Oct 30 21:47:46.659431 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.615033 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-cslfr"] Oct 30 21:47:46.659431 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.631118 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:46.659431 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.649386 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg"] Oct 30 21:47:46.659431 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.653162 5801 scope.go:122] "RemoveContainer" containerID="e63e393f18cc121add112014195a2b67d7f6c874d342e7b86314fe7bbbdf0237" Oct 30 21:47:46.659431 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:47:46.655802 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e63e393f18cc121add112014195a2b67d7f6c874d342e7b86314fe7bbbdf0237\": container with ID starting with e63e393f18cc121add112014195a2b67d7f6c874d342e7b86314fe7bbbdf0237 not found: ID does not exist" containerID="e63e393f18cc121add112014195a2b67d7f6c874d342e7b86314fe7bbbdf0237" Oct 30 21:47:46.659431 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.655852 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e63e393f18cc121add112014195a2b67d7f6c874d342e7b86314fe7bbbdf0237"} err="failed to get container status \"e63e393f18cc121add112014195a2b67d7f6c874d342e7b86314fe7bbbdf0237\": rpc error: code = NotFound desc = could not find container \"e63e393f18cc121add112014195a2b67d7f6c874d342e7b86314fe7bbbdf0237\": container with ID starting with e63e393f18cc121add112014195a2b67d7f6c874d342e7b86314fe7bbbdf0237 not found: ID does not exist" Oct 30 21:47:46.659431 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.655879 5801 scope.go:122] "RemoveContainer" containerID="816b1d8caa5e9ecaf74350b88519d18975c33a8c70a7f4fd8bbfe3c2d182cbec" Oct 30 21:47:46.679264 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.679219 5801 scope.go:122] "RemoveContainer" containerID="816b1d8caa5e9ecaf74350b88519d18975c33a8c70a7f4fd8bbfe3c2d182cbec" Oct 30 21:47:46.679693 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:47:46.679666 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"816b1d8caa5e9ecaf74350b88519d18975c33a8c70a7f4fd8bbfe3c2d182cbec\": container with ID starting with 816b1d8caa5e9ecaf74350b88519d18975c33a8c70a7f4fd8bbfe3c2d182cbec not found: ID does not exist" containerID="816b1d8caa5e9ecaf74350b88519d18975c33a8c70a7f4fd8bbfe3c2d182cbec" Oct 30 21:47:46.679848 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.679708 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"816b1d8caa5e9ecaf74350b88519d18975c33a8c70a7f4fd8bbfe3c2d182cbec"} err="failed to get container status \"816b1d8caa5e9ecaf74350b88519d18975c33a8c70a7f4fd8bbfe3c2d182cbec\": rpc error: code = NotFound desc = could not find container \"816b1d8caa5e9ecaf74350b88519d18975c33a8c70a7f4fd8bbfe3c2d182cbec\": container with ID starting with 816b1d8caa5e9ecaf74350b88519d18975c33a8c70a7f4fd8bbfe3c2d182cbec not found: ID does not exist" Oct 30 21:47:46.682875 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.682851 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-4wffg"] Oct 30 21:47:46.820389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:46.819935 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-833acf51-m7587" Oct 30 21:47:47.135546 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:47:47.135511 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda471144f_13fa_42dc_870a_7022db45e7db.slice/crio-cff1e0f0988035502fd34b567f402f55e7de166f876bb3b50e63cc0f467ce90b WatchSource:0}: Error finding container cff1e0f0988035502fd34b567f402f55e7de166f876bb3b50e63cc0f467ce90b: Status 404 returned error can't find the container with id cff1e0f0988035502fd34b567f402f55e7de166f876bb3b50e63cc0f467ce90b Oct 30 21:47:47.135888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:47.135862 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-833acf51-m7587"] Oct 30 21:47:47.472692 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:47.472626 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:47.532770 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:47.530029 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24b939ed-32e5-407e-a919-c0395d914dcd" path="/var/lib/kubelet/pods/24b939ed-32e5-407e-a919-c0395d914dcd/volumes" Oct 30 21:47:47.532770 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:47.530817 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="311939a8-e35d-4636-b052-a368f0a41a9a" path="/var/lib/kubelet/pods/311939a8-e35d-4636-b052-a368f0a41a9a/volumes" Oct 30 21:47:47.532770 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:47.531448 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bdabb86-13f1-4ab6-8b08-418fff19b933" path="/var/lib/kubelet/pods/7bdabb86-13f1-4ab6-8b08-418fff19b933/volumes" Oct 30 21:47:47.533928 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:47.533904 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-833acf51-m7587" event={"ID":"a471144f-13fa-42dc-870a-7022db45e7db","Type":"ContainerStarted","Data":"cff1e0f0988035502fd34b567f402f55e7de166f876bb3b50e63cc0f467ce90b"} Oct 30 21:47:48.677834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:48.675394 5801 generic.go:356] "Generic (PLEG): container finished" podID="a471144f-13fa-42dc-870a-7022db45e7db" containerID="97c9fde6445f086ab69966eb707ef92b6f6ddd225c14d41c40eef95273db0570" exitCode=0 Oct 30 21:47:48.677834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:48.675555 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-833acf51-m7587" event={"ID":"a471144f-13fa-42dc-870a-7022db45e7db","Type":"ContainerDied","Data":"97c9fde6445f086ab69966eb707ef92b6f6ddd225c14d41c40eef95273db0570"} Oct 30 21:47:48.770782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:48.762714 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:48.811795 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:48.805846 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:48.834809 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:48.827856 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:48.875790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:48.872957 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:49.236324 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:49.234929 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd" Oct 30 21:47:49.238934 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:49.238908 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j" Oct 30 21:47:49.265879 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:49.265834 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:49.287776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:49.266978 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:49.287776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:49.267021 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:49.287776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:49.284870 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:49.324836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:49.292734 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:49.451848 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:49.448895 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 21:47:49.470776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:49.467639 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:47:49.506839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:49.494918 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:47:49.593785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:49.591828 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:47:49.666775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:49.653399 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:49.666775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:49.653497 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:47:49.666775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:49.653897 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:47:49.769862 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:49.769580 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:49.769862 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:49.769814 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:49.898982 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:49.869443 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-448hk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:47:49.899718 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:49.899676 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" podUID="79d7a2c8-0585-48a3-88d4-d49b23672216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:49.999777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:49.967056 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:47:50.024832 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:50.008251 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:50.157779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:50.157323 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:47:50.157779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:50.157391 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:47:50.196777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:50.196374 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:50.285968 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:50.284462 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:50.285968 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:50.284843 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:50.285968 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:50.285083 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:47:50.285968 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:50.285144 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:47:50.288505 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:50.286889 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:50.366284 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:50.325884 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:47:50.456034 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:50.455987 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:50.975006 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:50.773845 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:47:50.975006 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:50.816102 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 30 21:47:50.975006 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:50.816439 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:47:50.975006 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:50.849512 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:47:50.975006 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:50.923045 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:47:50.975006 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:50.923164 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:47:50.975006 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:50.923286 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:47:50.975006 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:50.923358 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:47:50.975006 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:50.923411 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:47:50.988158 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:50.976203 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:47:50.988158 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:50.977009 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:47:51.129777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:51.114766 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded" Oct 30 21:47:51.276621 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:51.258940 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:51.339779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:51.267880 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:47:51.389939 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:51.304942 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:51.868470 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:51.585521 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" podUID="a8c67e3a-dafa-4ed4-b5e3-f1b341427687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Oct 30 21:47:51.888159 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:51.586358 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:51.888159 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:51.589916 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:51.888576 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:51.664345 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:47:51.888905 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:51.664600 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:51.963693 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:51.688953 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:47:52.044786 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.037257 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:47:52.044786 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:51.747608 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv" podUID="b6307469-5f7d-41d9-9490-e8381ff9bbfb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:52.044786 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:51.890965 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:47:52.044786 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.040777 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:47:52.044786 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:51.746467 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:47:52.166825 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.147356 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:52.166825 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.148687 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:47:52.220369 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.220337 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:52.264778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.260194 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:47:52.318695 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.318663 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-833acf51-m7587" Oct 30 21:47:52.420696 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.420604 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:47:52.420880 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.420702 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:47:52.420933 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.420860 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:47:52.420974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.420933 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:47:52.422191 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.422168 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:47:52.422333 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.422230 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6" Oct 30 21:47:52.422333 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.422293 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:47:52.422456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.422363 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:47:52.422456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.422404 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:47:52.422456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.422452 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:47:52.428060 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.428037 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg"] Oct 30 21:47:52.429193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.429173 5801 cpu_manager.go:414] "containerMap: removing container" podUID="a471144f-13fa-42dc-870a-7022db45e7db" containerName="revoke-keys" Oct 30 21:47:52.429193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.429192 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a471144f-13fa-42dc-870a-7022db45e7db" containerName="revoke-keys" Oct 30 21:47:52.429475 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.429457 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="a471144f-13fa-42dc-870a-7022db45e7db" containerName="revoke-keys" Oct 30 21:47:52.471473 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.471442 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg"] Oct 30 21:47:52.472492 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.472463 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.532054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.512858 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2ba0bd-openshift-default-dockercfg-5jlw4\"" Oct 30 21:47:52.532054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.529550 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a471144f-13fa-42dc-870a-7022db45e7db-kube-api-access-xfzgq\" (UniqueName: \"kubernetes.io/projected/a471144f-13fa-42dc-870a-7022db45e7db-kube-api-access-xfzgq\") pod \"a471144f-13fa-42dc-870a-7022db45e7db\" (UID: \"a471144f-13fa-42dc-870a-7022db45e7db\") " Oct 30 21:47:52.532054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.529639 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a471144f-13fa-42dc-870a-7022db45e7db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a471144f-13fa-42dc-870a-7022db45e7db-maas-api-openshift-ca\") pod \"a471144f-13fa-42dc-870a-7022db45e7db\" (UID: \"a471144f-13fa-42dc-870a-7022db45e7db\") " Oct 30 21:47:52.532054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.529816 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a471144f-13fa-42dc-870a-7022db45e7db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a471144f-13fa-42dc-870a-7022db45e7db-maas-api-xks-ca\") pod \"a471144f-13fa-42dc-870a-7022db45e7db\" (UID: \"a471144f-13fa-42dc-870a-7022db45e7db\") " Oct 30 21:47:52.532054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.531405 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a471144f-13fa-42dc-870a-7022db45e7db-maas-api-openshift-ca" pod "a471144f-13fa-42dc-870a-7022db45e7db" (UID: "a471144f-13fa-42dc-870a-7022db45e7db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:52.566070 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.551994 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a471144f-13fa-42dc-870a-7022db45e7db-kube-api-access-xfzgq" pod "a471144f-13fa-42dc-870a-7022db45e7db" (UID: "a471144f-13fa-42dc-870a-7022db45e7db"). InnerVolumeSpecName "kube-api-access-xfzgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:52.607693 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.566342 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a471144f-13fa-42dc-870a-7022db45e7db-maas-api-xks-ca" pod "a471144f-13fa-42dc-870a-7022db45e7db" (UID: "a471144f-13fa-42dc-870a-7022db45e7db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:52.607693 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.594563 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:52.638254 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.638195 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxmtp\" (UniqueName: \"kubernetes.io/projected/8690c196-570d-4604-bfee-2a5d84997f90-kube-api-access-wxmtp\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.638461 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.638336 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8690c196-570d-4604-bfee-2a5d84997f90-istio-token\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.638461 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.638440 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-workload-certs\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.638609 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.638574 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8690c196-570d-4604-bfee-2a5d84997f90-istio-podinfo\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.638673 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.638611 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-istio-envoy\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.638735 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.638700 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8690c196-570d-4604-bfee-2a5d84997f90-istiod-ca-cert\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.640054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.638890 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-credential-socket\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.640054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.638996 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-istio-data\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.640054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.639106 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-workload-socket\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.640054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.639276 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a471144f-13fa-42dc-870a-7022db45e7db-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:52.640054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.639296 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfzgq\" (UniqueName: \"kubernetes.io/projected/a471144f-13fa-42dc-870a-7022db45e7db-kube-api-access-xfzgq\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:52.640054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.639310 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a471144f-13fa-42dc-870a-7022db45e7db-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:52.739286 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.739197 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-833acf51-m7587" Oct 30 21:47:52.745940 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.745909 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-833acf51-m7587" event={"ID":"a471144f-13fa-42dc-870a-7022db45e7db","Type":"ContainerDied","Data":"cff1e0f0988035502fd34b567f402f55e7de166f876bb3b50e63cc0f467ce90b"} Oct 30 21:47:52.746167 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.746148 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cff1e0f0988035502fd34b567f402f55e7de166f876bb3b50e63cc0f467ce90b" Oct 30 21:47:52.752041 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.752015 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-credential-socket\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.752279 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.752257 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-istio-data\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.752442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.752422 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-workload-socket\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.752641 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.752623 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxmtp\" (UniqueName: \"kubernetes.io/projected/8690c196-570d-4604-bfee-2a5d84997f90-kube-api-access-wxmtp\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.768775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.752839 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8690c196-570d-4604-bfee-2a5d84997f90-istio-token\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.769446 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.769417 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-workload-certs\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.769699 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.769679 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8690c196-570d-4604-bfee-2a5d84997f90-istio-podinfo\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.769856 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.769838 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-istio-envoy\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.770042 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.770026 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8690c196-570d-4604-bfee-2a5d84997f90-istiod-ca-cert\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.798773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.758932 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-istio-data\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.799002 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.798075 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8690c196-570d-4604-bfee-2a5d84997f90-istio-podinfo\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.799119 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.798340 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-workload-certs\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.799199 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.759144 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-credential-socket\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.799953 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.799930 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8690c196-570d-4604-bfee-2a5d84997f90-istiod-ca-cert\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.811034 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.800697 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-istio-envoy\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.811181 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.759340 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-workload-socket\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.811261 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.807797 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8690c196-570d-4604-bfee-2a5d84997f90-istio-token\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:52.818907 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:52.818864 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxmtp\" (UniqueName: \"kubernetes.io/projected/8690c196-570d-4604-bfee-2a5d84997f90-kube-api-access-wxmtp\") pod \"e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:53.025235 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:53.025153 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:53.038550 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:53.038518 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0edc820-2shwf"] Oct 30 21:47:53.048958 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:53.048890 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0edc820-2shwf"] Oct 30 21:47:53.405479 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:53.405421 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg"] Oct 30 21:47:53.408777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:53.408715 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:47:53.409079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:53.409051 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:47:53.410038 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:53.410009 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:47:53.486926 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:53.486899 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:53.592779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:53.592363 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ffb13b5-2e3b-4f20-a091-a9f24155c339" path="/var/lib/kubelet/pods/2ffb13b5-2e3b-4f20-a091-a9f24155c339/volumes" Oct 30 21:47:53.752708 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:53.752626 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" event={"ID":"8690c196-570d-4604-bfee-2a5d84997f90","Type":"ContainerStarted","Data":"a2ee38c889b76a672392ba2971ab7582ffd557d721883da3214c4da9555829ce"} Oct 30 21:47:53.752708 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:53.752694 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" event={"ID":"8690c196-570d-4604-bfee-2a5d84997f90","Type":"ContainerStarted","Data":"a321b25b4f45c29101d29f2601ca34ded14c19f9fc0710229bdb24befb3e5af2"} Oct 30 21:47:53.788390 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:53.788325 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" podStartSLOduration=3.788308364 podStartE2EDuration="3.788308364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:53.78611523 +0000 UTC m=+1142.381932094" watchObservedRunningTime="2026-10-30 21:47:53.788308364 +0000 UTC m=+1142.384125025" Oct 30 21:47:54.026100 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:54.026021 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:54.027432 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:54.027407 5801 patch_prober.go:29] interesting pod/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 21:47:54.027558 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:54.027460 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" podUID="8690c196-570d-4604-bfee-2a5d84997f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 21:47:54.240048 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:54.240016 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:55.030369 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:55.030337 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:55.104580 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:55.104546 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:55.578196 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:55.578158 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:55.680775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:55.680255 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:55.805733 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:55.805682 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:55.806602 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:55.806578 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:47:56.190059 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:56.189972 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:56.363776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:56.325336 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:56.977370 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:56.977338 5801 scope.go:122] "RemoveContainer" containerID="9ef80080114e9814ff6fa12e462521682a6c7d788653e990a9d47b718ecfc1a3" Oct 30 21:47:56.988838 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:56.988815 5801 scope.go:122] "RemoveContainer" containerID="ed6ab96c86b45f59aaefd168e53ceca5b3fbdca3dc57bf4e4f1035c281c8b1d9" Oct 30 21:47:57.001985 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.001960 5801 scope.go:122] "RemoveContainer" containerID="723cbdb1ceb6d6f72257e39e51165a49c37e4cd9dc8c6f6e346a05736e37aa9d" Oct 30 21:47:57.012724 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.012704 5801 scope.go:122] "RemoveContainer" containerID="99592a71e1a630006caeb084d4ae81189497a4cb047dc13253f7fe9d15be2199" Oct 30 21:47:57.022709 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.022688 5801 scope.go:122] "RemoveContainer" containerID="906aa1b32975dd8fccf20c3e06703a1ac8b5c414bafb56941526b05cdeb7bb28" Oct 30 21:47:57.033233 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.033209 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7a0e4ed-88dm7"] Oct 30 21:47:57.034008 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.033987 5801 scope.go:122] "RemoveContainer" containerID="0168f3c69ee2d307553a5b261953c138f6c338ba3008126bc110a7603ca9ffbb" Oct 30 21:47:57.038169 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.038147 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7a0e4ed-88dm7"] Oct 30 21:47:57.044388 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.044369 5801 scope.go:122] "RemoveContainer" containerID="ba4b8c68dcc027aba5e433788cfc5703e540a1d4e01aff511fb48d146a433745" Oct 30 21:47:57.055096 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.055078 5801 scope.go:122] "RemoveContainer" containerID="60058831d0e778e3919cd2bfa9160fb72e3a15b4658c5dc371655a08aaccb024" Oct 30 21:47:57.064973 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.064953 5801 scope.go:122] "RemoveContainer" containerID="4e89804aa8bb4bb619f493a3e195853e0d6357a6c00154f79e110d44fed56a3e" Oct 30 21:47:57.075251 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.075231 5801 scope.go:122] "RemoveContainer" containerID="beddd0e617d1c48d8c320829ef983d8c070f05b53c9abfcb47f692c191510c8f" Oct 30 21:47:57.084881 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.084864 5801 scope.go:122] "RemoveContainer" containerID="5c00278fea947c0b190826252d0afbf34c26d256bb5ea7ad430b8a532c4be2d1" Oct 30 21:47:57.094297 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.094278 5801 scope.go:122] "RemoveContainer" containerID="6c21f90d21a6d01e3f9686c6d1be6e75d21d4fa0b43e94a2ffcfd65ea0b5129a" Oct 30 21:47:57.105512 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.105492 5801 scope.go:122] "RemoveContainer" containerID="6586bf5e49242be85cae4a8cd11d786ff7f021e3114d055347a38a1657e778bd" Oct 30 21:47:57.115153 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.115132 5801 scope.go:122] "RemoveContainer" containerID="f6bf890cb3b2e01c95326d09c9bf835073e258603071a396170221d35e34a3a9" Oct 30 21:47:57.126141 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.126124 5801 scope.go:122] "RemoveContainer" containerID="8b3384ffb9f510156233ceeddf5b2c50e5d47835fed2b6a5cdc9e4184c83f854" Oct 30 21:47:57.139447 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.139336 5801 scope.go:122] "RemoveContainer" containerID="c33be47b56c0a8073fe4d666eb40d9daf24a92ec2a2f7bc7c5385843701d7e4d" Oct 30 21:47:57.212852 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.212817 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:57.414792 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.410959 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:57.496197 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.496112 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7"] Oct 30 21:47:57.508828 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.508798 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7" Oct 30 21:47:57.524841 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.517377 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7"] Oct 30 21:47:57.525992 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.525964 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2ba0bd-dockercfg-62zt6\"" Oct 30 21:47:57.526271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.526247 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2ba0bd\"" Oct 30 21:47:57.533247 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.533215 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:57.554023 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.552744 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c7884e4-268e-46b2-808e-178ee8dc3398" path="/var/lib/kubelet/pods/5c7884e4-268e-46b2-808e-178ee8dc3398/volumes" Oct 30 21:47:57.554605 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.554580 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4"] Oct 30 21:47:57.567727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.567703 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4"] Oct 30 21:47:57.568036 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.568021 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4" Oct 30 21:47:57.797939 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.797847 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d03a2c0-4527-4104-ac16-dbadd1b59a97-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7\" (UID: \"9d03a2c0-4527-4104-ac16-dbadd1b59a97\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7" Oct 30 21:47:57.819468 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.819428 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nzh6\" (UniqueName: \"kubernetes.io/projected/9d03a2c0-4527-4104-ac16-dbadd1b59a97-kube-api-access-4nzh6\") pod \"payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7\" (UID: \"9d03a2c0-4527-4104-ac16-dbadd1b59a97\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7" Oct 30 21:47:57.927233 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.926872 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d03a2c0-4527-4104-ac16-dbadd1b59a97-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7\" (UID: \"9d03a2c0-4527-4104-ac16-dbadd1b59a97\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7" Oct 30 21:47:57.927233 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.926952 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nzh6\" (UniqueName: \"kubernetes.io/projected/9d03a2c0-4527-4104-ac16-dbadd1b59a97-kube-api-access-4nzh6\") pod \"payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7\" (UID: \"9d03a2c0-4527-4104-ac16-dbadd1b59a97\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7" Oct 30 21:47:57.929086 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.927624 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtmm7\" (UniqueName: \"kubernetes.io/projected/951263d7-9273-48d6-9f89-e60bf70b7998-kube-api-access-dtmm7\") pod \"payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4\" (UID: \"951263d7-9273-48d6-9f89-e60bf70b7998\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4" Oct 30 21:47:57.929086 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.927697 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951263d7-9273-48d6-9f89-e60bf70b7998-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4\" (UID: \"951263d7-9273-48d6-9f89-e60bf70b7998\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4" Oct 30 21:47:57.930409 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.929551 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d03a2c0-4527-4104-ac16-dbadd1b59a97-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7\" (UID: \"9d03a2c0-4527-4104-ac16-dbadd1b59a97\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7" Oct 30 21:47:57.931897 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.931871 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:57.956577 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:57.956545 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nzh6\" (UniqueName: \"kubernetes.io/projected/9d03a2c0-4527-4104-ac16-dbadd1b59a97-kube-api-access-4nzh6\") pod \"payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7\" (UID: \"9d03a2c0-4527-4104-ac16-dbadd1b59a97\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7" Oct 30 21:47:58.030206 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:58.030162 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtmm7\" (UniqueName: \"kubernetes.io/projected/951263d7-9273-48d6-9f89-e60bf70b7998-kube-api-access-dtmm7\") pod \"payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4\" (UID: \"951263d7-9273-48d6-9f89-e60bf70b7998\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4" Oct 30 21:47:58.030720 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:58.030253 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951263d7-9273-48d6-9f89-e60bf70b7998-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4\" (UID: \"951263d7-9273-48d6-9f89-e60bf70b7998\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4" Oct 30 21:47:58.031094 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:58.031072 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951263d7-9273-48d6-9f89-e60bf70b7998-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4\" (UID: \"951263d7-9273-48d6-9f89-e60bf70b7998\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4" Oct 30 21:47:58.062931 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:58.059832 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtmm7\" (UniqueName: \"kubernetes.io/projected/951263d7-9273-48d6-9f89-e60bf70b7998-kube-api-access-dtmm7\") pod \"payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4\" (UID: \"951263d7-9273-48d6-9f89-e60bf70b7998\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4" Oct 30 21:47:58.099067 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:58.096993 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4" Oct 30 21:47:58.178325 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:58.178290 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7" Oct 30 21:47:58.232361 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:58.231883 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:58.423867 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:47:58.423832 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951263d7_9273_48d6_9f89_e60bf70b7998.slice/crio-aaefe7c4fa60d7afb22636704775d2505e7cbd7f4e1a86ff8d300e3038851afd WatchSource:0}: Error finding container aaefe7c4fa60d7afb22636704775d2505e7cbd7f4e1a86ff8d300e3038851afd: Status 404 returned error can't find the container with id aaefe7c4fa60d7afb22636704775d2505e7cbd7f4e1a86ff8d300e3038851afd Oct 30 21:47:58.438970 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:58.438942 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4"] Oct 30 21:47:58.534708 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:47:58.534535 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d03a2c0_4527_4104_ac16_dbadd1b59a97.slice/crio-d57a282fe7e388f1d49eb3e50e24b8e4afe6a572697f1b563ce8e64f97dbd0a0 WatchSource:0}: Error finding container d57a282fe7e388f1d49eb3e50e24b8e4afe6a572697f1b563ce8e64f97dbd0a0: Status 404 returned error can't find the container with id d57a282fe7e388f1d49eb3e50e24b8e4afe6a572697f1b563ce8e64f97dbd0a0 Oct 30 21:47:58.534998 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:58.534967 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7"] Oct 30 21:47:58.881670 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:58.881313 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4" event={"ID":"951263d7-9273-48d6-9f89-e60bf70b7998","Type":"ContainerStarted","Data":"5a1829a60e66a5e11864e107b300811366745ec867acf01c1f734c7a69b2d330"} Oct 30 21:47:58.881670 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:58.881376 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4" event={"ID":"951263d7-9273-48d6-9f89-e60bf70b7998","Type":"ContainerStarted","Data":"aaefe7c4fa60d7afb22636704775d2505e7cbd7f4e1a86ff8d300e3038851afd"} Oct 30 21:47:58.881670 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:58.881410 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4" Oct 30 21:47:58.883625 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:58.883593 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7" event={"ID":"9d03a2c0-4527-4104-ac16-dbadd1b59a97","Type":"ContainerStarted","Data":"033820a601dfbf7b22f31904915e2d391d22452c7ffae1573a4dd87d2153b601"} Oct 30 21:47:58.883777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:58.883647 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7" event={"ID":"9d03a2c0-4527-4104-ac16-dbadd1b59a97","Type":"ContainerStarted","Data":"d57a282fe7e388f1d49eb3e50e24b8e4afe6a572697f1b563ce8e64f97dbd0a0"} Oct 30 21:47:58.883777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:58.883680 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7" Oct 30 21:47:58.904488 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:58.904443 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4" podStartSLOduration=1.9044301940000001 podStartE2EDuration="1.904430194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:58.901658205 +0000 UTC m=+1147.497474865" watchObservedRunningTime="2026-10-30 21:47:58.904430194 +0000 UTC m=+1147.500246853" Oct 30 21:47:58.920938 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:58.920898 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7" podStartSLOduration=1.920886246 podStartE2EDuration="1.920886246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:58.918933268 +0000 UTC m=+1147.514749918" watchObservedRunningTime="2026-10-30 21:47:58.920886246 +0000 UTC m=+1147.516702907" Oct 30 21:47:58.954999 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:58.954941 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:59.727496 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:47:59.727458 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:00.280456 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:48:00.280417 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:48:00.412828 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:00.412794 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:00.911466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:00.911383 5801 generic.go:356] "Generic (PLEG): container finished" podID="3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e" containerID="cbab4eddbdbb21198e1d111ef24547a637bb3ca3648360505872d2cb402dffe9" exitCode=0 Oct 30 21:48:00.911466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:00.911424 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" event={"ID":"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e","Type":"ContainerDied","Data":"cbab4eddbdbb21198e1d111ef24547a637bb3ca3648360505872d2cb402dffe9"} Oct 30 21:48:00.911986 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:00.911849 5801 scope.go:122] "RemoveContainer" containerID="cbab4eddbdbb21198e1d111ef24547a637bb3ca3648360505872d2cb402dffe9" Oct 30 21:48:00.913576 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:00.913546 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:48:00.913641 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:00.913617 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:48:00.913682 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:00.913645 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:48:01.163264 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:01.163170 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:01.897352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:01.897307 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:01.923313 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:01.923284 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" event={"ID":"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e","Type":"ContainerStarted","Data":"2234d70e1ea909adb643e94f3f7348c408d4b02151880f068245e2d826f44fc2"} Oct 30 21:48:02.576236 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:02.574925 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:48:02.577972 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:02.577941 5801 patch_prober.go:29] interesting pod/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 21:48:02.578125 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:02.578000 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" podUID="3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 21:48:02.604315 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:02.604284 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:03.301655 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:03.301622 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:03.578492 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:03.578138 5801 patch_prober.go:29] interesting pod/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 21:48:03.578492 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:03.578208 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" podUID="3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 21:48:04.020677 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:04.020579 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:04.208344 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:04.208215 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:04.372651 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:04.372610 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:04.383654 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:04.383622 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m"] Oct 30 21:48:04.429835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:04.422850 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-jnm8m"] Oct 30 21:48:04.497573 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:04.497538 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:04.572201 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:04.572163 5801 patch_prober.go:29] interesting pod/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 21:48:04.572502 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:04.572474 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" podUID="3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 21:48:05.584330 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:05.584292 5801 patch_prober.go:29] interesting pod/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 21:48:05.586489 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:05.586452 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" podUID="3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 21:48:05.637517 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:05.637482 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="192d484c-6d1c-4d8f-ab93-7f9d011317e7" path="/var/lib/kubelet/pods/192d484c-6d1c-4d8f-ab93-7f9d011317e7/volumes" Oct 30 21:48:05.638328 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:05.638305 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:48:05.793022 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:05.792957 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:06.552436 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:06.552404 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:06.575774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:06.575057 5801 patch_prober.go:29] interesting pod/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 21:48:06.575774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:06.575119 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" podUID="3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 21:48:07.323092 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:07.323042 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:07.654365 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:07.654269 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:48:07.655822 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:07.655796 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:48:08.239230 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:08.239189 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:08.779906 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:08.779874 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:09.132121 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:09.132076 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:09.531204 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:09.531125 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl"] Oct 30 21:48:09.531383 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:09.531341 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" podUID="8d851d22-81c2-4ba2-90b7-e5d82d1b0929" containerName="istio-proxy" containerID="cri-o://12eeaf9980be6d48efb4f17b3c2419661a59fdf0cb17830db2d63f150605f466" gracePeriod=30 Oct 30 21:48:09.899714 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:09.899680 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4" Oct 30 21:48:09.900135 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:09.899744 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7" Oct 30 21:48:09.925860 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:09.925831 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:10.341617 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:10.341583 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:10.651457 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:10.651372 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:11.055248 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.055206 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:11.571962 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.571926 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5"] Oct 30 21:48:11.576536 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.576516 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.598063 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.598032 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5"] Oct 30 21:48:11.620567 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.620536 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9975b4c-127e-4130-ae64-b8785b0acc7f-istiod-ca-cert\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.620742 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.620667 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-token\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.620742 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.620726 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-podinfo\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.620910 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.620808 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-workload-certs\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.620973 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.620933 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-workload-socket\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.621032 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.621018 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9g8z\" (UniqueName: \"kubernetes.io/projected/f9975b4c-127e-4130-ae64-b8785b0acc7f-kube-api-access-k9g8z\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.621137 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.621118 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-credential-socket\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.621212 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.621195 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-envoy\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.621810 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.621330 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-data\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.631135 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.631103 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:11.645288 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.645263 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-950784-openshift-default-dockercfg-72v9p\"" Oct 30 21:48:11.739774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.730156 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-workload-socket\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.739774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.730271 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9g8z\" (UniqueName: \"kubernetes.io/projected/f9975b4c-127e-4130-ae64-b8785b0acc7f-kube-api-access-k9g8z\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.739774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.730370 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-credential-socket\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.739774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.730443 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-envoy\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.739774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.730574 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-data\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.739774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.730637 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9975b4c-127e-4130-ae64-b8785b0acc7f-istiod-ca-cert\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.739774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.730719 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-token\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.739774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.730775 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-podinfo\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.739774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.730817 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-workload-certs\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.739774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.731384 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-workload-certs\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.739774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.731629 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-workload-socket\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.739774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.732220 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-credential-socket\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.739774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.738471 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9975b4c-127e-4130-ae64-b8785b0acc7f-istiod-ca-cert\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.739774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.738743 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-data\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.765782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.761118 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-envoy\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.785344 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.782977 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:11.790980 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.783815 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9g8z\" (UniqueName: \"kubernetes.io/projected/f9975b4c-127e-4130-ae64-b8785b0acc7f-kube-api-access-k9g8z\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.806455 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.806423 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-podinfo\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.831831 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.830507 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-token\") pod \"e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:11.896123 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:11.896082 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:12.018247 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:12.015827 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:12.136504 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:12.135744 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:12.238832 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:12.237618 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:12.398684 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:48:12.398655 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9975b4c_127e_4130_ae64_b8785b0acc7f.slice/crio-1517e292fd5667b2c5882e433e5b4f5e78c0f2c0d1c9fdfd5383ceb2afa9d890 WatchSource:0}: Error finding container 1517e292fd5667b2c5882e433e5b4f5e78c0f2c0d1c9fdfd5383ceb2afa9d890: Status 404 returned error can't find the container with id 1517e292fd5667b2c5882e433e5b4f5e78c0f2c0d1c9fdfd5383ceb2afa9d890 Oct 30 21:48:12.399717 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:12.399691 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5"] Oct 30 21:48:12.401640 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:12.401608 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:48:12.401714 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:12.401676 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:48:12.401714 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:12.401706 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:48:12.730545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:12.730444 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:12.836023 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:12.835966 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:12.840037 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:12.840005 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8"] Oct 30 21:48:12.845351 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:12.845328 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8" Oct 30 21:48:12.852521 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:12.852496 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-08aafeac-dockercfg-v7b48\"" Oct 30 21:48:12.853172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:12.853150 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-08aafeac\"" Oct 30 21:48:12.857927 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:12.856096 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8"] Oct 30 21:48:12.963787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:12.963363 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbtk5\" (UniqueName: \"kubernetes.io/projected/6d340f73-5c9e-49d7-8703-5043db3c6193-kube-api-access-jbtk5\") pod \"payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8\" (UID: \"6d340f73-5c9e-49d7-8703-5043db3c6193\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8" Oct 30 21:48:12.963787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:12.963483 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d340f73-5c9e-49d7-8703-5043db3c6193-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8\" (UID: \"6d340f73-5c9e-49d7-8703-5043db3c6193\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8" Oct 30 21:48:13.088776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:13.079408 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbtk5\" (UniqueName: \"kubernetes.io/projected/6d340f73-5c9e-49d7-8703-5043db3c6193-kube-api-access-jbtk5\") pod \"payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8\" (UID: \"6d340f73-5c9e-49d7-8703-5043db3c6193\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8" Oct 30 21:48:13.088776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:13.079507 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d340f73-5c9e-49d7-8703-5043db3c6193-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8\" (UID: \"6d340f73-5c9e-49d7-8703-5043db3c6193\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8" Oct 30 21:48:13.088776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:13.080965 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d340f73-5c9e-49d7-8703-5043db3c6193-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8\" (UID: \"6d340f73-5c9e-49d7-8703-5043db3c6193\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8" Oct 30 21:48:13.126407 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:13.122080 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbtk5\" (UniqueName: \"kubernetes.io/projected/6d340f73-5c9e-49d7-8703-5043db3c6193-kube-api-access-jbtk5\") pod \"payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8\" (UID: \"6d340f73-5c9e-49d7-8703-5043db3c6193\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8" Oct 30 21:48:13.186302 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:13.185053 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8" Oct 30 21:48:13.212651 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:13.211416 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" event={"ID":"f9975b4c-127e-4130-ae64-b8785b0acc7f","Type":"ContainerStarted","Data":"cb9ad72496a0cd9cbd1dd57911db06b323ed7484e3511285d9020bed8a079349"} Oct 30 21:48:13.212651 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:13.211502 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" event={"ID":"f9975b4c-127e-4130-ae64-b8785b0acc7f","Type":"ContainerStarted","Data":"1517e292fd5667b2c5882e433e5b4f5e78c0f2c0d1c9fdfd5383ceb2afa9d890"} Oct 30 21:48:13.246037 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:13.245986 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" podStartSLOduration=2.245971086 podStartE2EDuration="2.245971086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:13.239691055 +0000 UTC m=+1161.835507730" watchObservedRunningTime="2026-10-30 21:48:13.245971086 +0000 UTC m=+1161.841787745" Oct 30 21:48:13.403672 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:13.403583 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:13.516770 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:13.516722 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8"] Oct 30 21:48:13.520950 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:48:13.520919 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d340f73_5c9e_49d7_8703_5043db3c6193.slice/crio-469fa407371979c515993134becedf22d4a9891c3acc948a110620ad61a528f7 WatchSource:0}: Error finding container 469fa407371979c515993134becedf22d4a9891c3acc948a110620ad61a528f7: Status 404 returned error can't find the container with id 469fa407371979c515993134becedf22d4a9891c3acc948a110620ad61a528f7 Oct 30 21:48:13.897635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:13.897604 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:13.904731 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:13.904699 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:14.222895 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.222857 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8" event={"ID":"6d340f73-5c9e-49d7-8703-5043db3c6193","Type":"ContainerStarted","Data":"d27197f7ab67ad7eb68692965d4d5fd8eb7eaf070db6ba407fb3afa96d2b9979"} Oct 30 21:48:14.223352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.222924 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8" event={"ID":"6d340f73-5c9e-49d7-8703-5043db3c6193","Type":"ContainerStarted","Data":"469fa407371979c515993134becedf22d4a9891c3acc948a110620ad61a528f7"} Oct 30 21:48:14.223352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.223294 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8" Oct 30 21:48:14.223352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.223342 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:14.224091 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.224071 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:48:14.244687 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.244647 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8" podStartSLOduration=2.244635569 podStartE2EDuration="2.244635569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:14.243021614 +0000 UTC m=+1162.838838293" watchObservedRunningTime="2026-10-30 21:48:14.244635569 +0000 UTC m=+1162.840452229" Oct 30 21:48:14.455263 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.455229 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:14.823636 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.823611 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:48:14.924411 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.924374 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-workload-socket\") pod \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " Oct 30 21:48:14.924584 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.924499 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istiod-ca-cert\") pod \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " Oct 30 21:48:14.924658 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.924642 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-envoy\") pod \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " Oct 30 21:48:14.924705 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.924644 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-workload-socket" pod "8d851d22-81c2-4ba2-90b7-e5d82d1b0929" (UID: "8d851d22-81c2-4ba2-90b7-e5d82d1b0929"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:14.924705 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.924686 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-token\" (UniqueName: \"kubernetes.io/projected/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-token\") pod \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " Oct 30 21:48:14.924812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.924788 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-workload-certs\") pod \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " Oct 30 21:48:14.924862 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.924848 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-credential-socket\") pod \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " Oct 30 21:48:14.924908 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.924876 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-podinfo\") pod \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " Oct 30 21:48:14.924908 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.924884 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istiod-ca-cert" pod "8d851d22-81c2-4ba2-90b7-e5d82d1b0929" (UID: "8d851d22-81c2-4ba2-90b7-e5d82d1b0929"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:14.925002 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.924908 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-data\") pod \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " Oct 30 21:48:14.925002 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.924937 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-kube-api-access-mcm4b\" (UniqueName: \"kubernetes.io/projected/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-kube-api-access-mcm4b\") pod \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\" (UID: \"8d851d22-81c2-4ba2-90b7-e5d82d1b0929\") " Oct 30 21:48:14.925095 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.925053 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-workload-certs" pod "8d851d22-81c2-4ba2-90b7-e5d82d1b0929" (UID: "8d851d22-81c2-4ba2-90b7-e5d82d1b0929"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:14.925315 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.925158 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-credential-socket" pod "8d851d22-81c2-4ba2-90b7-e5d82d1b0929" (UID: "8d851d22-81c2-4ba2-90b7-e5d82d1b0929"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:14.925315 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.925232 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-data" pod "8d851d22-81c2-4ba2-90b7-e5d82d1b0929" (UID: "8d851d22-81c2-4ba2-90b7-e5d82d1b0929"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:14.925907 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.925810 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:14.925907 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.925839 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:14.925907 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.925854 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:14.925907 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.925869 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:14.925907 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.925883 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:14.927094 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.927073 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-podinfo" pod "8d851d22-81c2-4ba2-90b7-e5d82d1b0929" (UID: "8d851d22-81c2-4ba2-90b7-e5d82d1b0929"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:48:14.927212 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.927194 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-envoy" pod "8d851d22-81c2-4ba2-90b7-e5d82d1b0929" (UID: "8d851d22-81c2-4ba2-90b7-e5d82d1b0929"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:14.927538 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.927519 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-kube-api-access-mcm4b" pod "8d851d22-81c2-4ba2-90b7-e5d82d1b0929" (UID: "8d851d22-81c2-4ba2-90b7-e5d82d1b0929"). InnerVolumeSpecName "kube-api-access-mcm4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:14.927620 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:14.927583 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-token" pod "8d851d22-81c2-4ba2-90b7-e5d82d1b0929" (UID: "8d851d22-81c2-4ba2-90b7-e5d82d1b0929"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:15.027704 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:15.027664 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:15.027704 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:15.027696 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:15.027704 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:15.027707 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:15.027962 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:15.027716 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcm4b\" (UniqueName: \"kubernetes.io/projected/8d851d22-81c2-4ba2-90b7-e5d82d1b0929-kube-api-access-mcm4b\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:15.233625 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:15.233545 5801 generic.go:356] "Generic (PLEG): container finished" podID="8d851d22-81c2-4ba2-90b7-e5d82d1b0929" containerID="12eeaf9980be6d48efb4f17b3c2419661a59fdf0cb17830db2d63f150605f466" exitCode=0 Oct 30 21:48:15.233625 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:15.233591 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" Oct 30 21:48:15.234139 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:15.233631 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" event={"ID":"8d851d22-81c2-4ba2-90b7-e5d82d1b0929","Type":"ContainerDied","Data":"12eeaf9980be6d48efb4f17b3c2419661a59fdf0cb17830db2d63f150605f466"} Oct 30 21:48:15.234139 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:15.233709 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl" event={"ID":"8d851d22-81c2-4ba2-90b7-e5d82d1b0929","Type":"ContainerDied","Data":"8c3e92e8fe913414284fb24c63c906a2edc8656af6cc42e3f7b29bacc35e924e"} Oct 30 21:48:15.234139 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:15.233731 5801 scope.go:122] "RemoveContainer" containerID="12eeaf9980be6d48efb4f17b3c2419661a59fdf0cb17830db2d63f150605f466" Oct 30 21:48:15.248118 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:15.248101 5801 scope.go:122] "RemoveContainer" containerID="12eeaf9980be6d48efb4f17b3c2419661a59fdf0cb17830db2d63f150605f466" Oct 30 21:48:15.248447 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:48:15.248419 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12eeaf9980be6d48efb4f17b3c2419661a59fdf0cb17830db2d63f150605f466\": container with ID starting with 12eeaf9980be6d48efb4f17b3c2419661a59fdf0cb17830db2d63f150605f466 not found: ID does not exist" containerID="12eeaf9980be6d48efb4f17b3c2419661a59fdf0cb17830db2d63f150605f466" Oct 30 21:48:15.248528 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:15.248452 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12eeaf9980be6d48efb4f17b3c2419661a59fdf0cb17830db2d63f150605f466"} err="failed to get container status \"12eeaf9980be6d48efb4f17b3c2419661a59fdf0cb17830db2d63f150605f466\": rpc error: code = NotFound desc = could not find container \"12eeaf9980be6d48efb4f17b3c2419661a59fdf0cb17830db2d63f150605f466\": container with ID starting with 12eeaf9980be6d48efb4f17b3c2419661a59fdf0cb17830db2d63f150605f466 not found: ID does not exist" Oct 30 21:48:15.264082 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:15.264057 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl"] Oct 30 21:48:15.268716 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:15.268696 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-kcwbl"] Oct 30 21:48:15.481960 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:15.481161 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:15.552518 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:15.552419 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d851d22-81c2-4ba2-90b7-e5d82d1b0929" path="/var/lib/kubelet/pods/8d851d22-81c2-4ba2-90b7-e5d82d1b0929/volumes" Oct 30 21:48:15.985493 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:15.985457 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:16.148846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:16.144284 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:16.897255 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:16.897216 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:17.323851 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.323815 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:17.367283 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.367255 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6"] Oct 30 21:48:17.368066 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.368050 5801 cpu_manager.go:414] "containerMap: removing container" podUID="8d851d22-81c2-4ba2-90b7-e5d82d1b0929" containerName="istio-proxy" Oct 30 21:48:17.368120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.368067 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d851d22-81c2-4ba2-90b7-e5d82d1b0929" containerName="istio-proxy" Oct 30 21:48:17.368198 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.368187 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d851d22-81c2-4ba2-90b7-e5d82d1b0929" containerName="istio-proxy" Oct 30 21:48:17.374193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.374173 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6" Oct 30 21:48:17.389416 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.389398 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-950784\"" Oct 30 21:48:17.389549 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.389532 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-950784-dockercfg-22c78\"" Oct 30 21:48:17.405550 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.405478 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6"] Oct 30 21:48:17.431797 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.431714 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz"] Oct 30 21:48:17.439972 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.439939 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz" Oct 30 21:48:17.447697 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.447653 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz"] Oct 30 21:48:17.552787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.516621 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efeccb5-6c34-4aa7-9470-9611f088505f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6\" (UID: \"8efeccb5-6c34-4aa7-9470-9611f088505f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6" Oct 30 21:48:17.552787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.516740 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nzkh\" (UniqueName: \"kubernetes.io/projected/8efeccb5-6c34-4aa7-9470-9611f088505f-kube-api-access-2nzkh\") pod \"payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6\" (UID: \"8efeccb5-6c34-4aa7-9470-9611f088505f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6" Oct 30 21:48:17.552787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.516800 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f3e0a6-905e-46f6-a273-da4ee1e0e3f2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz\" (UID: \"92f3e0a6-905e-46f6-a273-da4ee1e0e3f2\") " pod="openshift-ingress/payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz" Oct 30 21:48:17.556950 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.556645 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69wfz\" (UniqueName: \"kubernetes.io/projected/92f3e0a6-905e-46f6-a273-da4ee1e0e3f2-kube-api-access-69wfz\") pod \"payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz\" (UID: \"92f3e0a6-905e-46f6-a273-da4ee1e0e3f2\") " pod="openshift-ingress/payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz" Oct 30 21:48:17.645873 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.637430 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:17.668795 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.663159 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69wfz\" (UniqueName: \"kubernetes.io/projected/92f3e0a6-905e-46f6-a273-da4ee1e0e3f2-kube-api-access-69wfz\") pod \"payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz\" (UID: \"92f3e0a6-905e-46f6-a273-da4ee1e0e3f2\") " pod="openshift-ingress/payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz" Oct 30 21:48:17.668795 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.663486 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efeccb5-6c34-4aa7-9470-9611f088505f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6\" (UID: \"8efeccb5-6c34-4aa7-9470-9611f088505f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6" Oct 30 21:48:17.668795 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.663541 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nzkh\" (UniqueName: \"kubernetes.io/projected/8efeccb5-6c34-4aa7-9470-9611f088505f-kube-api-access-2nzkh\") pod \"payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6\" (UID: \"8efeccb5-6c34-4aa7-9470-9611f088505f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6" Oct 30 21:48:17.668795 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.663571 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f3e0a6-905e-46f6-a273-da4ee1e0e3f2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz\" (UID: \"92f3e0a6-905e-46f6-a273-da4ee1e0e3f2\") " pod="openshift-ingress/payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz" Oct 30 21:48:17.668795 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.664622 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f3e0a6-905e-46f6-a273-da4ee1e0e3f2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz\" (UID: \"92f3e0a6-905e-46f6-a273-da4ee1e0e3f2\") " pod="openshift-ingress/payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz" Oct 30 21:48:17.668795 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.665721 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efeccb5-6c34-4aa7-9470-9611f088505f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6\" (UID: \"8efeccb5-6c34-4aa7-9470-9611f088505f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6" Oct 30 21:48:17.748923 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.748878 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69wfz\" (UniqueName: \"kubernetes.io/projected/92f3e0a6-905e-46f6-a273-da4ee1e0e3f2-kube-api-access-69wfz\") pod \"payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz\" (UID: \"92f3e0a6-905e-46f6-a273-da4ee1e0e3f2\") " pod="openshift-ingress/payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz" Oct 30 21:48:17.771799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.770935 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz" Oct 30 21:48:17.772588 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.772554 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nzkh\" (UniqueName: \"kubernetes.io/projected/8efeccb5-6c34-4aa7-9470-9611f088505f-kube-api-access-2nzkh\") pod \"payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6\" (UID: \"8efeccb5-6c34-4aa7-9470-9611f088505f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6" Oct 30 21:48:17.993968 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:17.990813 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6" Oct 30 21:48:18.082276 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:18.082235 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:18.227671 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:48:18.227593 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f3e0a6_905e_46f6_a273_da4ee1e0e3f2.slice/crio-53fdca7e455d1f0506e831b8cc7c65128944388ab95a4bc0a06bbf8347732b92 WatchSource:0}: Error finding container 53fdca7e455d1f0506e831b8cc7c65128944388ab95a4bc0a06bbf8347732b92: Status 404 returned error can't find the container with id 53fdca7e455d1f0506e831b8cc7c65128944388ab95a4bc0a06bbf8347732b92 Oct 30 21:48:18.236901 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:18.236609 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz"] Oct 30 21:48:18.273690 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:18.273657 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7386906d-wp74z"] Oct 30 21:48:18.280632 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:18.280611 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7386906d-wp74z" Oct 30 21:48:18.285976 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:18.285954 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:48:18.285976 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:18.285963 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:48:18.287224 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:18.287105 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7386906d-wp74z"] Oct 30 21:48:18.287304 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:18.287266 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:48:18.287373 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:18.287357 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:48:18.325159 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:18.324137 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6"] Oct 30 21:48:18.326195 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:18.326154 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz" event={"ID":"92f3e0a6-905e-46f6-a273-da4ee1e0e3f2","Type":"ContainerStarted","Data":"53fdca7e455d1f0506e831b8cc7c65128944388ab95a4bc0a06bbf8347732b92"} Oct 30 21:48:18.327478 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:48:18.327453 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8efeccb5_6c34_4aa7_9470_9611f088505f.slice/crio-305cc7a2fce70ad2684503a3bcb872aa6df6400d7dfd87fcbe9bfcb976ce25f7 WatchSource:0}: Error finding container 305cc7a2fce70ad2684503a3bcb872aa6df6400d7dfd87fcbe9bfcb976ce25f7: Status 404 returned error can't find the container with id 305cc7a2fce70ad2684503a3bcb872aa6df6400d7dfd87fcbe9bfcb976ce25f7 Oct 30 21:48:18.405907 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:18.405878 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8vdd\" (UniqueName: \"kubernetes.io/projected/11886179-f8d2-4237-a992-8103eed3ad7c-kube-api-access-v8vdd\") pod \"maas-api-revoke-keys-e2e-mt-7386906d-wp74z\" (UID: \"11886179-f8d2-4237-a992-8103eed3ad7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7386906d-wp74z" Oct 30 21:48:18.406213 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:18.406195 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11886179-f8d2-4237-a992-8103eed3ad7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7386906d-wp74z\" (UID: \"11886179-f8d2-4237-a992-8103eed3ad7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7386906d-wp74z" Oct 30 21:48:18.406379 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:18.406363 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11886179-f8d2-4237-a992-8103eed3ad7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7386906d-wp74z\" (UID: \"11886179-f8d2-4237-a992-8103eed3ad7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7386906d-wp74z" Oct 30 21:48:18.507673 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:18.507605 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11886179-f8d2-4237-a992-8103eed3ad7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7386906d-wp74z\" (UID: \"11886179-f8d2-4237-a992-8103eed3ad7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7386906d-wp74z" Oct 30 21:48:18.507827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:18.507714 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11886179-f8d2-4237-a992-8103eed3ad7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7386906d-wp74z\" (UID: \"11886179-f8d2-4237-a992-8103eed3ad7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7386906d-wp74z" Oct 30 21:48:18.507827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:18.507737 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8vdd\" (UniqueName: \"kubernetes.io/projected/11886179-f8d2-4237-a992-8103eed3ad7c-kube-api-access-v8vdd\") pod \"maas-api-revoke-keys-e2e-mt-7386906d-wp74z\" (UID: \"11886179-f8d2-4237-a992-8103eed3ad7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7386906d-wp74z" Oct 30 21:48:18.508244 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:18.508224 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11886179-f8d2-4237-a992-8103eed3ad7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7386906d-wp74z\" (UID: \"11886179-f8d2-4237-a992-8103eed3ad7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7386906d-wp74z" Oct 30 21:48:18.509980 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:18.509962 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11886179-f8d2-4237-a992-8103eed3ad7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7386906d-wp74z\" (UID: \"11886179-f8d2-4237-a992-8103eed3ad7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7386906d-wp74z" Oct 30 21:48:18.520518 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:18.520496 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8vdd\" (UniqueName: \"kubernetes.io/projected/11886179-f8d2-4237-a992-8103eed3ad7c-kube-api-access-v8vdd\") pod \"maas-api-revoke-keys-e2e-mt-7386906d-wp74z\" (UID: \"11886179-f8d2-4237-a992-8103eed3ad7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7386906d-wp74z" Oct 30 21:48:18.608002 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:18.607976 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7386906d-wp74z" Oct 30 21:48:18.903036 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:18.903007 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7386906d-wp74z"] Oct 30 21:48:18.903397 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:48:18.903375 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11886179_f8d2_4237_a992_8103eed3ad7c.slice/crio-7308f57495bfb00b4c4d4122a51aee16af77a53787eee4c263c5ee1b89e9f8ff WatchSource:0}: Error finding container 7308f57495bfb00b4c4d4122a51aee16af77a53787eee4c263c5ee1b89e9f8ff: Status 404 returned error can't find the container with id 7308f57495bfb00b4c4d4122a51aee16af77a53787eee4c263c5ee1b89e9f8ff Oct 30 21:48:19.046285 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:19.046247 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:19.337234 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:19.337198 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6" event={"ID":"8efeccb5-6c34-4aa7-9470-9611f088505f","Type":"ContainerStarted","Data":"fab7978257ff4ecd92529284d01d8d04ab5c9fb0f2a5ab41f28de052e82c586a"} Oct 30 21:48:19.337413 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:19.337257 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6" event={"ID":"8efeccb5-6c34-4aa7-9470-9611f088505f","Type":"ContainerStarted","Data":"305cc7a2fce70ad2684503a3bcb872aa6df6400d7dfd87fcbe9bfcb976ce25f7"} Oct 30 21:48:19.337413 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:19.337284 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6" Oct 30 21:48:19.338702 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:19.338674 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7386906d-wp74z" event={"ID":"11886179-f8d2-4237-a992-8103eed3ad7c","Type":"ContainerStarted","Data":"7308f57495bfb00b4c4d4122a51aee16af77a53787eee4c263c5ee1b89e9f8ff"} Oct 30 21:48:19.340429 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:19.340404 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz" event={"ID":"92f3e0a6-905e-46f6-a273-da4ee1e0e3f2","Type":"ContainerStarted","Data":"2a3ad39d38ac25dc5fd24059ff73f064df98a83f48134de290ef47ec1ff2c0d0"} Oct 30 21:48:19.340549 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:19.340483 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz" Oct 30 21:48:19.362277 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:19.362238 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6" podStartSLOduration=2.362224125 podStartE2EDuration="2.362224125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:19.355729115 +0000 UTC m=+1167.951545794" watchObservedRunningTime="2026-10-30 21:48:19.362224125 +0000 UTC m=+1167.958040784" Oct 30 21:48:19.377578 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:19.377539 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz" podStartSLOduration=2.377524529 podStartE2EDuration="2.377524529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:19.375938612 +0000 UTC m=+1167.971755292" watchObservedRunningTime="2026-10-30 21:48:19.377524529 +0000 UTC m=+1167.973341189" Oct 30 21:48:19.785618 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:19.785579 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:20.353208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:20.353135 5801 generic.go:356] "Generic (PLEG): container finished" podID="11886179-f8d2-4237-a992-8103eed3ad7c" containerID="ea3fa7099be1f01db676fcaa9bbf08af47a4968c2cce659adb171b7a2efbcd8d" exitCode=0 Oct 30 21:48:20.353603 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:20.353221 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7386906d-wp74z" event={"ID":"11886179-f8d2-4237-a992-8103eed3ad7c","Type":"ContainerDied","Data":"ea3fa7099be1f01db676fcaa9bbf08af47a4968c2cce659adb171b7a2efbcd8d"} Oct 30 21:48:20.591262 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:20.591224 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:21.402272 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:21.402237 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:21.601462 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:21.601435 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7386906d-wp74z" Oct 30 21:48:21.692869 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:21.692792 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11886179-f8d2-4237-a992-8103eed3ad7c-kube-api-access-v8vdd\" (UniqueName: \"kubernetes.io/projected/11886179-f8d2-4237-a992-8103eed3ad7c-kube-api-access-v8vdd\") pod \"11886179-f8d2-4237-a992-8103eed3ad7c\" (UID: \"11886179-f8d2-4237-a992-8103eed3ad7c\") " Oct 30 21:48:21.693017 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:21.692891 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11886179-f8d2-4237-a992-8103eed3ad7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11886179-f8d2-4237-a992-8103eed3ad7c-maas-api-openshift-ca\") pod \"11886179-f8d2-4237-a992-8103eed3ad7c\" (UID: \"11886179-f8d2-4237-a992-8103eed3ad7c\") " Oct 30 21:48:21.693096 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:21.693080 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11886179-f8d2-4237-a992-8103eed3ad7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11886179-f8d2-4237-a992-8103eed3ad7c-maas-api-xks-ca\") pod \"11886179-f8d2-4237-a992-8103eed3ad7c\" (UID: \"11886179-f8d2-4237-a992-8103eed3ad7c\") " Oct 30 21:48:21.693269 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:21.693242 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11886179-f8d2-4237-a992-8103eed3ad7c-maas-api-openshift-ca" pod "11886179-f8d2-4237-a992-8103eed3ad7c" (UID: "11886179-f8d2-4237-a992-8103eed3ad7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:21.693903 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:21.693879 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11886179-f8d2-4237-a992-8103eed3ad7c-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:21.695355 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:21.695332 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11886179-f8d2-4237-a992-8103eed3ad7c-kube-api-access-v8vdd" pod "11886179-f8d2-4237-a992-8103eed3ad7c" (UID: "11886179-f8d2-4237-a992-8103eed3ad7c"). InnerVolumeSpecName "kube-api-access-v8vdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:21.695446 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:21.695346 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11886179-f8d2-4237-a992-8103eed3ad7c-maas-api-xks-ca" pod "11886179-f8d2-4237-a992-8103eed3ad7c" (UID: "11886179-f8d2-4237-a992-8103eed3ad7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:21.795600 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:21.795570 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8vdd\" (UniqueName: \"kubernetes.io/projected/11886179-f8d2-4237-a992-8103eed3ad7c-kube-api-access-v8vdd\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:21.795600 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:21.795599 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11886179-f8d2-4237-a992-8103eed3ad7c-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:22.153799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:22.153693 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:22.376000 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:22.375968 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7386906d-wp74z" Oct 30 21:48:22.376172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:22.376006 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7386906d-wp74z" event={"ID":"11886179-f8d2-4237-a992-8103eed3ad7c","Type":"ContainerDied","Data":"7308f57495bfb00b4c4d4122a51aee16af77a53787eee4c263c5ee1b89e9f8ff"} Oct 30 21:48:22.376172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:22.376095 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7308f57495bfb00b4c4d4122a51aee16af77a53787eee4c263c5ee1b89e9f8ff" Oct 30 21:48:22.857253 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:22.857223 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:23.555476 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:23.555445 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:23.663206 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:23.663169 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:23.860826 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:23.856868 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:24.512316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:24.512282 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:25.237972 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:25.237940 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8" Oct 30 21:48:25.409552 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:25.409520 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4"] Oct 30 21:48:25.409763 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:25.409731 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" podUID="6aaf74ef-1550-443a-a046-d08fc45e07f1" containerName="istio-proxy" containerID="cri-o://69b10569b4e2f336e6c7eea1aa64f4d05674b0081cb3afd71472129b19405761" gracePeriod=30 Oct 30 21:48:25.419897 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:25.419850 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h"] Oct 30 21:48:25.420097 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:25.420068 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" podUID="0f59fa92-8480-42ba-b9e7-424c3b05b8e2" containerName="istio-proxy" containerID="cri-o://1d7ae2f796f1490b0d04c2872a1b5c2c742673b4bba79935541de8dd02c164b1" gracePeriod=30 Oct 30 21:48:25.428565 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:25.428538 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn"] Oct 30 21:48:25.428789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:25.428736 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" podUID="321186ea-7520-430c-825b-b8129b422d95" containerName="istio-proxy" containerID="cri-o://470c2ceb080097ee36a8e3448c13b630cc6f4d2eb0fd654fa88e919cae426264" gracePeriod=30 Oct 30 21:48:25.465850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:25.465785 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:26.290290 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:26.290256 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:26.584646 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:26.584614 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:48:26.584821 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:26.584664 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" podUID="321186ea-7520-430c-825b-b8129b422d95" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:48:27.085740 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:27.083296 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:27.749844 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:27.749808 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:27.924833 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:27.924793 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:48:27.924984 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:27.924857 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" podUID="6aaf74ef-1550-443a-a046-d08fc45e07f1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:48:28.382929 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:28.382897 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7386906d-78c5b4547c-84v44"] Oct 30 21:48:28.398925 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:28.398899 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq"] Oct 30 21:48:28.543470 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:28.543430 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:28.646391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:28.646203 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:29.443233 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:29.443198 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:30.162020 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.161980 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:30.284110 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:48:30.284073 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:48:30.356296 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.356269 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6" Oct 30 21:48:30.356475 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.356341 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz" Oct 30 21:48:30.559459 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.559430 5801 generic.go:356] "Generic (PLEG): container finished" podID="0f59fa92-8480-42ba-b9e7-424c3b05b8e2" containerID="1d7ae2f796f1490b0d04c2872a1b5c2c742673b4bba79935541de8dd02c164b1" exitCode=0 Oct 30 21:48:30.559839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.559604 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" event={"ID":"0f59fa92-8480-42ba-b9e7-424c3b05b8e2","Type":"ContainerDied","Data":"1d7ae2f796f1490b0d04c2872a1b5c2c742673b4bba79935541de8dd02c164b1"} Oct 30 21:48:30.562046 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.562017 5801 generic.go:356] "Generic (PLEG): container finished" podID="6aaf74ef-1550-443a-a046-d08fc45e07f1" containerID="69b10569b4e2f336e6c7eea1aa64f4d05674b0081cb3afd71472129b19405761" exitCode=0 Oct 30 21:48:30.562214 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.562197 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" event={"ID":"6aaf74ef-1550-443a-a046-d08fc45e07f1","Type":"ContainerDied","Data":"69b10569b4e2f336e6c7eea1aa64f4d05674b0081cb3afd71472129b19405761"} Oct 30 21:48:30.564117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.564092 5801 generic.go:356] "Generic (PLEG): container finished" podID="321186ea-7520-430c-825b-b8129b422d95" containerID="470c2ceb080097ee36a8e3448c13b630cc6f4d2eb0fd654fa88e919cae426264" exitCode=0 Oct 30 21:48:30.564261 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.564243 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" event={"ID":"321186ea-7520-430c-825b-b8129b422d95","Type":"ContainerDied","Data":"470c2ceb080097ee36a8e3448c13b630cc6f4d2eb0fd654fa88e919cae426264"} Oct 30 21:48:30.784774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.784690 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:48:30.844787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.844743 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:48:30.848862 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.848835 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:48:30.859659 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.859630 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-token\" (UniqueName: \"kubernetes.io/projected/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-token\") pod \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " Oct 30 21:48:30.859838 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.859794 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-istio-data\") pod \"321186ea-7520-430c-825b-b8129b422d95\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " Oct 30 21:48:30.859913 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.859874 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321186ea-7520-430c-825b-b8129b422d95-istio-token\" (UniqueName: \"kubernetes.io/projected/321186ea-7520-430c-825b-b8129b422d95-istio-token\") pod \"321186ea-7520-430c-825b-b8129b422d95\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " Oct 30 21:48:30.859996 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.859943 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-credential-socket\") pod \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " Oct 30 21:48:30.859996 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.859973 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-workload-socket\") pod \"6aaf74ef-1550-443a-a046-d08fc45e07f1\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " Oct 30 21:48:30.860115 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.860005 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/321186ea-7520-430c-825b-b8129b422d95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321186ea-7520-430c-825b-b8129b422d95-istio-podinfo\") pod \"321186ea-7520-430c-825b-b8129b422d95\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " Oct 30 21:48:30.860115 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.860043 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-envoy\") pod \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " Oct 30 21:48:30.860115 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.860108 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-workload-certs\") pod \"321186ea-7520-430c-825b-b8129b422d95\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " Oct 30 21:48:30.860271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.860153 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-credential-socket\") pod \"321186ea-7520-430c-825b-b8129b422d95\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " Oct 30 21:48:30.860271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.860161 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-istio-data" pod "321186ea-7520-430c-825b-b8129b422d95" (UID: "321186ea-7520-430c-825b-b8129b422d95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.860271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.860197 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-data\") pod \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " Oct 30 21:48:30.860271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.860247 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-workload-socket\") pod \"321186ea-7520-430c-825b-b8129b422d95\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " Oct 30 21:48:30.860472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.860304 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-token\" (UniqueName: \"kubernetes.io/projected/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-token\") pod \"6aaf74ef-1550-443a-a046-d08fc45e07f1\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " Oct 30 21:48:30.860472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.860345 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-podinfo\") pod \"6aaf74ef-1550-443a-a046-d08fc45e07f1\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " Oct 30 21:48:30.860472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.860373 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-workload-socket" pod "6aaf74ef-1550-443a-a046-d08fc45e07f1" (UID: "6aaf74ef-1550-443a-a046-d08fc45e07f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.860472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.860383 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-podinfo\") pod \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " Oct 30 21:48:30.860472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.860423 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-workload-certs\") pod \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " Oct 30 21:48:30.860472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.860452 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/321186ea-7520-430c-825b-b8129b422d95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321186ea-7520-430c-825b-b8129b422d95-istiod-ca-cert\") pod \"321186ea-7520-430c-825b-b8129b422d95\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " Oct 30 21:48:30.860799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.860493 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istiod-ca-cert\") pod \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " Oct 30 21:48:30.860799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.860520 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-kube-api-access-vd2st\" (UniqueName: \"kubernetes.io/projected/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-kube-api-access-vd2st\") pod \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " Oct 30 21:48:30.860799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.860569 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aaf74ef-1550-443a-a046-d08fc45e07f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aaf74ef-1550-443a-a046-d08fc45e07f1-istiod-ca-cert\") pod \"6aaf74ef-1550-443a-a046-d08fc45e07f1\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " Oct 30 21:48:30.860799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.860634 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321186ea-7520-430c-825b-b8129b422d95-kube-api-access-nk5rw\" (UniqueName: \"kubernetes.io/projected/321186ea-7520-430c-825b-b8129b422d95-kube-api-access-nk5rw\") pod \"321186ea-7520-430c-825b-b8129b422d95\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " Oct 30 21:48:30.860799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.860657 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-credential-socket" pod "0f59fa92-8480-42ba-b9e7-424c3b05b8e2" (UID: "0f59fa92-8480-42ba-b9e7-424c3b05b8e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.860799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.860661 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-istio-envoy\") pod \"321186ea-7520-430c-825b-b8129b422d95\" (UID: \"321186ea-7520-430c-825b-b8129b422d95\") " Oct 30 21:48:30.860799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.860733 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-workload-certs\") pod \"6aaf74ef-1550-443a-a046-d08fc45e07f1\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " Oct 30 21:48:30.861251 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.860843 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-envoy\") pod \"6aaf74ef-1550-443a-a046-d08fc45e07f1\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " Oct 30 21:48:30.861251 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.860904 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-workload-socket\") pod \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\" (UID: \"0f59fa92-8480-42ba-b9e7-424c3b05b8e2\") " Oct 30 21:48:30.861251 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.861176 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-workload-socket" pod "0f59fa92-8480-42ba-b9e7-424c3b05b8e2" (UID: "0f59fa92-8480-42ba-b9e7-424c3b05b8e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.861420 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.861367 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-workload-certs" pod "6aaf74ef-1550-443a-a046-d08fc45e07f1" (UID: "6aaf74ef-1550-443a-a046-d08fc45e07f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.861673 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.861645 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-credential-socket" pod "321186ea-7520-430c-825b-b8129b422d95" (UID: "321186ea-7520-430c-825b-b8129b422d95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.861795 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.861713 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aaf74ef-1550-443a-a046-d08fc45e07f1-istiod-ca-cert" pod "6aaf74ef-1550-443a-a046-d08fc45e07f1" (UID: "6aaf74ef-1550-443a-a046-d08fc45e07f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:30.862360 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.862320 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-token" pod "0f59fa92-8480-42ba-b9e7-424c3b05b8e2" (UID: "0f59fa92-8480-42ba-b9e7-424c3b05b8e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:30.862505 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.862385 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-workload-certs" pod "321186ea-7520-430c-825b-b8129b422d95" (UID: "321186ea-7520-430c-825b-b8129b422d95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.863101 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.862663 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-data" pod "0f59fa92-8480-42ba-b9e7-424c3b05b8e2" (UID: "0f59fa92-8480-42ba-b9e7-424c3b05b8e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.863101 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.862859 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/321186ea-7520-430c-825b-b8129b422d95-istiod-ca-cert" pod "321186ea-7520-430c-825b-b8129b422d95" (UID: "321186ea-7520-430c-825b-b8129b422d95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:30.863925 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.863781 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-workload-socket" pod "321186ea-7520-430c-825b-b8129b422d95" (UID: "321186ea-7520-430c-825b-b8129b422d95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.863925 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.863897 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321186ea-7520-430c-825b-b8129b422d95-istio-token" pod "321186ea-7520-430c-825b-b8129b422d95" (UID: "321186ea-7520-430c-825b-b8129b422d95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:30.864125 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.864090 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aaf74ef-1550-443a-a046-d08fc45e07f1-kube-api-access-rrpwb\" (UniqueName: \"kubernetes.io/projected/6aaf74ef-1550-443a-a046-d08fc45e07f1-kube-api-access-rrpwb\") pod \"6aaf74ef-1550-443a-a046-d08fc45e07f1\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " Oct 30 21:48:30.864511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.864223 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-credential-socket\") pod \"6aaf74ef-1550-443a-a046-d08fc45e07f1\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " Oct 30 21:48:30.864511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.864248 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-workload-certs" pod "0f59fa92-8480-42ba-b9e7-424c3b05b8e2" (UID: "0f59fa92-8480-42ba-b9e7-424c3b05b8e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.864511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.864263 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-data\") pod \"6aaf74ef-1550-443a-a046-d08fc45e07f1\" (UID: \"6aaf74ef-1550-443a-a046-d08fc45e07f1\") " Oct 30 21:48:30.865261 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.865234 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/321186ea-7520-430c-825b-b8129b422d95-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.865352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.865266 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.865352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.865281 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.865352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.865295 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.865352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.865309 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.865352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.865323 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.865352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.865335 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.865352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.865350 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.865838 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.865364 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321186ea-7520-430c-825b-b8129b422d95-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.865838 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.865378 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aaf74ef-1550-443a-a046-d08fc45e07f1-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.865838 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.865393 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.865838 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.865405 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.865838 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.865419 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.865838 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.865432 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.865838 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.865592 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/321186ea-7520-430c-825b-b8129b422d95-istio-podinfo" pod "321186ea-7520-430c-825b-b8129b422d95" (UID: "321186ea-7520-430c-825b-b8129b422d95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:48:30.865838 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.865829 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istiod-ca-cert" pod "0f59fa92-8480-42ba-b9e7-424c3b05b8e2" (UID: "0f59fa92-8480-42ba-b9e7-424c3b05b8e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:30.866273 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.865838 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-podinfo" pod "6aaf74ef-1550-443a-a046-d08fc45e07f1" (UID: "6aaf74ef-1550-443a-a046-d08fc45e07f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:48:30.866273 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.865863 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-credential-socket" pod "6aaf74ef-1550-443a-a046-d08fc45e07f1" (UID: "6aaf74ef-1550-443a-a046-d08fc45e07f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.866273 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.865898 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-data" pod "6aaf74ef-1550-443a-a046-d08fc45e07f1" (UID: "6aaf74ef-1550-443a-a046-d08fc45e07f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.867953 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.867926 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321186ea-7520-430c-825b-b8129b422d95-kube-api-access-nk5rw" pod "321186ea-7520-430c-825b-b8129b422d95" (UID: "321186ea-7520-430c-825b-b8129b422d95"). InnerVolumeSpecName "kube-api-access-nk5rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:30.868081 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.868057 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-envoy" pod "0f59fa92-8480-42ba-b9e7-424c3b05b8e2" (UID: "0f59fa92-8480-42ba-b9e7-424c3b05b8e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.868176 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.868159 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aaf74ef-1550-443a-a046-d08fc45e07f1-kube-api-access-rrpwb" pod "6aaf74ef-1550-443a-a046-d08fc45e07f1" (UID: "6aaf74ef-1550-443a-a046-d08fc45e07f1"). InnerVolumeSpecName "kube-api-access-rrpwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:30.868234 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.868165 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-podinfo" pod "0f59fa92-8480-42ba-b9e7-424c3b05b8e2" (UID: "0f59fa92-8480-42ba-b9e7-424c3b05b8e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:48:30.869142 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.869087 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-envoy" pod "6aaf74ef-1550-443a-a046-d08fc45e07f1" (UID: "6aaf74ef-1550-443a-a046-d08fc45e07f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.869142 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.869111 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-token" pod "6aaf74ef-1550-443a-a046-d08fc45e07f1" (UID: "6aaf74ef-1550-443a-a046-d08fc45e07f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:30.869142 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.869104 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-istio-envoy" pod "321186ea-7520-430c-825b-b8129b422d95" (UID: "321186ea-7520-430c-825b-b8129b422d95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:30.870359 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.870338 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-kube-api-access-vd2st" pod "0f59fa92-8480-42ba-b9e7-424c3b05b8e2" (UID: "0f59fa92-8480-42ba-b9e7-424c3b05b8e2"). InnerVolumeSpecName "kube-api-access-vd2st". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:30.966938 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.966904 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321186ea-7520-430c-825b-b8129b422d95-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.966938 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.966933 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.966938 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.966943 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.967166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.966953 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.967166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.966962 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.967166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.966970 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.967166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.966980 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd2st\" (UniqueName: \"kubernetes.io/projected/0f59fa92-8480-42ba-b9e7-424c3b05b8e2-kube-api-access-vd2st\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.967166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.966990 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk5rw\" (UniqueName: \"kubernetes.io/projected/321186ea-7520-430c-825b-b8129b422d95-kube-api-access-nk5rw\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.967166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.966999 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321186ea-7520-430c-825b-b8129b422d95-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.967166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.967007 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.967166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.967015 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrpwb\" (UniqueName: \"kubernetes.io/projected/6aaf74ef-1550-443a-a046-d08fc45e07f1-kube-api-access-rrpwb\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.967166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.967024 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:30.967166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:30.967032 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aaf74ef-1550-443a-a046-d08fc45e07f1-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:31.062195 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:31.062162 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:31.573993 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:31.573969 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" Oct 30 21:48:31.574337 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:31.574004 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4" event={"ID":"6aaf74ef-1550-443a-a046-d08fc45e07f1","Type":"ContainerDied","Data":"8eaf73101a498d2568e3a90dc769ff86debb32a1296fbd41e3ca48c3a9e8889f"} Oct 30 21:48:31.574337 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:31.574061 5801 scope.go:122] "RemoveContainer" containerID="69b10569b4e2f336e6c7eea1aa64f4d05674b0081cb3afd71472129b19405761" Oct 30 21:48:31.576619 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:31.576130 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" Oct 30 21:48:31.576619 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:31.576146 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn" event={"ID":"321186ea-7520-430c-825b-b8129b422d95","Type":"ContainerDied","Data":"98eae10d7ea186e38f155762da2539fda0da794649c49fa9f5e6aaebab8b8287"} Oct 30 21:48:31.578152 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:31.577870 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" event={"ID":"0f59fa92-8480-42ba-b9e7-424c3b05b8e2","Type":"ContainerDied","Data":"84895d7861824fc870bfeaedce7b4adde85c9646116774d1b317cb8dad0177e2"} Oct 30 21:48:31.578152 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:31.577956 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h" Oct 30 21:48:31.587209 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:31.587188 5801 scope.go:122] "RemoveContainer" containerID="470c2ceb080097ee36a8e3448c13b630cc6f4d2eb0fd654fa88e919cae426264" Oct 30 21:48:31.597332 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:31.597310 5801 scope.go:122] "RemoveContainer" containerID="1d7ae2f796f1490b0d04c2872a1b5c2c742673b4bba79935541de8dd02c164b1" Oct 30 21:48:31.604907 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:31.604871 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h"] Oct 30 21:48:31.612595 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:31.612575 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-gb92h"] Oct 30 21:48:31.621565 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:31.621544 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn"] Oct 30 21:48:31.629954 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:31.629925 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-f5ptn"] Oct 30 21:48:31.642082 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:31.642058 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4"] Oct 30 21:48:31.646330 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:31.646309 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-xfgh4"] Oct 30 21:48:32.003775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:32.003212 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:32.857488 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:32.857336 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:33.384088 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:33.384051 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7386906d-78c5b4547c-84v44" podUID="30476067-bd3b-4eac-825e-19fc7879eed8" containerName="payload-processing" containerID="cri-o://e75b09e74f8898948f607b64f1b3b66ab2f503907245464f8043cbfb65c448f8" gracePeriod=25 Oct 30 21:48:33.410603 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:33.405535 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq" podUID="76f0a594-7906-4abe-bb8f-a4265394e9ce" containerName="payload-pre-processing" containerID="cri-o://fb38dcf70dcba431793c7546ef8dd7c92f4ff9795af0e4400ad25ffb9b3be367" gracePeriod=25 Oct 30 21:48:33.635011 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:33.634476 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f59fa92-8480-42ba-b9e7-424c3b05b8e2" path="/var/lib/kubelet/pods/0f59fa92-8480-42ba-b9e7-424c3b05b8e2/volumes" Oct 30 21:48:33.635365 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:33.635339 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="321186ea-7520-430c-825b-b8129b422d95" path="/var/lib/kubelet/pods/321186ea-7520-430c-825b-b8129b422d95/volumes" Oct 30 21:48:33.676275 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:33.676245 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aaf74ef-1550-443a-a046-d08fc45e07f1" path="/var/lib/kubelet/pods/6aaf74ef-1550-443a-a046-d08fc45e07f1/volumes" Oct 30 21:48:33.677098 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:33.677060 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7386906d-78c5b4547c-84v44" event={"ID":"30476067-bd3b-4eac-825e-19fc7879eed8","Type":"ContainerDied","Data":"e75b09e74f8898948f607b64f1b3b66ab2f503907245464f8043cbfb65c448f8"} Oct 30 21:48:33.677304 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:33.673473 5801 generic.go:356] "Generic (PLEG): container finished" podID="30476067-bd3b-4eac-825e-19fc7879eed8" containerID="e75b09e74f8898948f607b64f1b3b66ab2f503907245464f8043cbfb65c448f8" exitCode=0 Oct 30 21:48:33.692922 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:33.687217 5801 generic.go:356] "Generic (PLEG): container finished" podID="76f0a594-7906-4abe-bb8f-a4265394e9ce" containerID="fb38dcf70dcba431793c7546ef8dd7c92f4ff9795af0e4400ad25ffb9b3be367" exitCode=0 Oct 30 21:48:33.692922 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:33.687281 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq" event={"ID":"76f0a594-7906-4abe-bb8f-a4265394e9ce","Type":"ContainerDied","Data":"fb38dcf70dcba431793c7546ef8dd7c92f4ff9795af0e4400ad25ffb9b3be367"} Oct 30 21:48:33.922323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:33.922247 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7386906d-78c5b4547c-84v44" Oct 30 21:48:33.954686 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:33.954647 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:33.955773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:33.955732 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30476067-bd3b-4eac-825e-19fc7879eed8-kube-api-access-jdvjm\" (UniqueName: \"kubernetes.io/projected/30476067-bd3b-4eac-825e-19fc7879eed8-kube-api-access-jdvjm\") pod \"30476067-bd3b-4eac-825e-19fc7879eed8\" (UID: \"30476067-bd3b-4eac-825e-19fc7879eed8\") " Oct 30 21:48:33.956176 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:33.956153 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30476067-bd3b-4eac-825e-19fc7879eed8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30476067-bd3b-4eac-825e-19fc7879eed8-plugins-config-volume\") pod \"30476067-bd3b-4eac-825e-19fc7879eed8\" (UID: \"30476067-bd3b-4eac-825e-19fc7879eed8\") " Oct 30 21:48:33.956544 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:33.956519 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30476067-bd3b-4eac-825e-19fc7879eed8-plugins-config-volume" pod "30476067-bd3b-4eac-825e-19fc7879eed8" (UID: "30476067-bd3b-4eac-825e-19fc7879eed8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:33.957481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:33.957455 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30476067-bd3b-4eac-825e-19fc7879eed8-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:33.957964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:33.957944 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30476067-bd3b-4eac-825e-19fc7879eed8-kube-api-access-jdvjm" pod "30476067-bd3b-4eac-825e-19fc7879eed8" (UID: "30476067-bd3b-4eac-825e-19fc7879eed8"). InnerVolumeSpecName "kube-api-access-jdvjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:33.976717 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:33.976686 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq" Oct 30 21:48:34.060985 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:34.060418 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76f0a594-7906-4abe-bb8f-a4265394e9ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76f0a594-7906-4abe-bb8f-a4265394e9ce-plugins-config-volume\") pod \"76f0a594-7906-4abe-bb8f-a4265394e9ce\" (UID: \"76f0a594-7906-4abe-bb8f-a4265394e9ce\") " Oct 30 21:48:34.060985 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:34.060626 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76f0a594-7906-4abe-bb8f-a4265394e9ce-kube-api-access-t44j9\" (UniqueName: \"kubernetes.io/projected/76f0a594-7906-4abe-bb8f-a4265394e9ce-kube-api-access-t44j9\") pod \"76f0a594-7906-4abe-bb8f-a4265394e9ce\" (UID: \"76f0a594-7906-4abe-bb8f-a4265394e9ce\") " Oct 30 21:48:34.066827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:34.061471 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdvjm\" (UniqueName: \"kubernetes.io/projected/30476067-bd3b-4eac-825e-19fc7879eed8-kube-api-access-jdvjm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:34.093788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:34.092457 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:34.093788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:34.093125 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76f0a594-7906-4abe-bb8f-a4265394e9ce-plugins-config-volume" pod "76f0a594-7906-4abe-bb8f-a4265394e9ce" (UID: "76f0a594-7906-4abe-bb8f-a4265394e9ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:34.098240 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:34.098210 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f0a594-7906-4abe-bb8f-a4265394e9ce-kube-api-access-t44j9" pod "76f0a594-7906-4abe-bb8f-a4265394e9ce" (UID: "76f0a594-7906-4abe-bb8f-a4265394e9ce"). InnerVolumeSpecName "kube-api-access-t44j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:34.163545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:34.163510 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76f0a594-7906-4abe-bb8f-a4265394e9ce-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:34.163545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:34.163541 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t44j9\" (UniqueName: \"kubernetes.io/projected/76f0a594-7906-4abe-bb8f-a4265394e9ce-kube-api-access-t44j9\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:34.698168 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:34.698084 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7386906d-78c5b4547c-84v44" Oct 30 21:48:34.698168 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:34.698098 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7386906d-78c5b4547c-84v44" event={"ID":"30476067-bd3b-4eac-825e-19fc7879eed8","Type":"ContainerDied","Data":"78fc2ae9689f788b940045d845a58036d5d9cfdd0520b402bd9f145dad136c4f"} Oct 30 21:48:34.698437 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:34.698174 5801 scope.go:122] "RemoveContainer" containerID="e75b09e74f8898948f607b64f1b3b66ab2f503907245464f8043cbfb65c448f8" Oct 30 21:48:34.700087 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:34.700060 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq" event={"ID":"76f0a594-7906-4abe-bb8f-a4265394e9ce","Type":"ContainerDied","Data":"9a89ba98dd1a9fac16136b7010ccb600d87f597a7ae6423eb73fb941d5d1979f"} Oct 30 21:48:34.700205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:34.700167 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq" Oct 30 21:48:34.712540 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:34.712525 5801 scope.go:122] "RemoveContainer" containerID="fb38dcf70dcba431793c7546ef8dd7c92f4ff9795af0e4400ad25ffb9b3be367" Oct 30 21:48:34.730743 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:34.730716 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq"] Oct 30 21:48:34.735449 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:34.735430 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7386906d-f8fd988cf-r6tlq"] Oct 30 21:48:34.747021 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:34.747001 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7386906d-78c5b4547c-84v44"] Oct 30 21:48:34.751187 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:34.751162 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7386906d-78c5b4547c-84v44"] Oct 30 21:48:34.772475 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:34.772435 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:35.526671 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:35.526637 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30476067-bd3b-4eac-825e-19fc7879eed8" path="/var/lib/kubelet/pods/30476067-bd3b-4eac-825e-19fc7879eed8/volumes" Oct 30 21:48:35.527256 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:35.527235 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76f0a594-7906-4abe-bb8f-a4265394e9ce" path="/var/lib/kubelet/pods/76f0a594-7906-4abe-bb8f-a4265394e9ce/volumes" Oct 30 21:48:35.777835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:35.774447 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:36.636790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:36.636733 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:36.907668 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:36.907585 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:37.427308 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:37.427271 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:37.569776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:37.569086 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:37.713952 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:37.713833 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:37.719436 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:37.719379 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7386906d-wp74z"] Oct 30 21:48:37.747278 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:37.747247 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7386906d-wp74z"] Oct 30 21:48:37.833066 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:37.832900 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:38.267029 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:38.266545 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:38.817119 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:38.817083 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:39.383499 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.383459 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr"] Oct 30 21:48:39.384941 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.384907 5801 cpu_manager.go:414] "containerMap: removing container" podUID="30476067-bd3b-4eac-825e-19fc7879eed8" containerName="payload-processing" Oct 30 21:48:39.384941 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.384936 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30476067-bd3b-4eac-825e-19fc7879eed8" containerName="payload-processing" Oct 30 21:48:39.385089 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.384950 5801 cpu_manager.go:414] "containerMap: removing container" podUID="11886179-f8d2-4237-a992-8103eed3ad7c" containerName="revoke-keys" Oct 30 21:48:39.385089 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.384976 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11886179-f8d2-4237-a992-8103eed3ad7c" containerName="revoke-keys" Oct 30 21:48:39.385089 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.384996 5801 cpu_manager.go:414] "containerMap: removing container" podUID="0f59fa92-8480-42ba-b9e7-424c3b05b8e2" containerName="istio-proxy" Oct 30 21:48:39.385089 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.385004 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f59fa92-8480-42ba-b9e7-424c3b05b8e2" containerName="istio-proxy" Oct 30 21:48:39.385089 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.385021 5801 cpu_manager.go:414] "containerMap: removing container" podUID="6aaf74ef-1550-443a-a046-d08fc45e07f1" containerName="istio-proxy" Oct 30 21:48:39.385089 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.385028 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aaf74ef-1550-443a-a046-d08fc45e07f1" containerName="istio-proxy" Oct 30 21:48:39.385271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.385112 5801 cpu_manager.go:414] "containerMap: removing container" podUID="76f0a594-7906-4abe-bb8f-a4265394e9ce" containerName="payload-pre-processing" Oct 30 21:48:39.385271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.385120 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76f0a594-7906-4abe-bb8f-a4265394e9ce" containerName="payload-pre-processing" Oct 30 21:48:39.385271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.385134 5801 cpu_manager.go:414] "containerMap: removing container" podUID="321186ea-7520-430c-825b-b8129b422d95" containerName="istio-proxy" Oct 30 21:48:39.385271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.385143 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321186ea-7520-430c-825b-b8129b422d95" containerName="istio-proxy" Oct 30 21:48:39.385393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.385311 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="76f0a594-7906-4abe-bb8f-a4265394e9ce" containerName="payload-pre-processing" Oct 30 21:48:39.385393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.385325 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="321186ea-7520-430c-825b-b8129b422d95" containerName="istio-proxy" Oct 30 21:48:39.385393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.385336 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="30476067-bd3b-4eac-825e-19fc7879eed8" containerName="payload-processing" Oct 30 21:48:39.385393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.385344 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="11886179-f8d2-4237-a992-8103eed3ad7c" containerName="revoke-keys" Oct 30 21:48:39.385393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.385356 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f59fa92-8480-42ba-b9e7-424c3b05b8e2" containerName="istio-proxy" Oct 30 21:48:39.385393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.385366 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aaf74ef-1550-443a-a046-d08fc45e07f1" containerName="istio-proxy" Oct 30 21:48:39.391250 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.391224 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr"] Oct 30 21:48:39.391367 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.391347 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr" Oct 30 21:48:39.395557 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.395519 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:48:39.396530 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.396507 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:48:39.418320 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.418292 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:48:39.418448 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.418389 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:48:39.519014 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.518983 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbc85c6d-b1d2-405d-b3cf-de42924b33e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr\" (UID: \"bbc85c6d-b1d2-405d-b3cf-de42924b33e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr" Oct 30 21:48:39.519251 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.519228 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbc85c6d-b1d2-405d-b3cf-de42924b33e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr\" (UID: \"bbc85c6d-b1d2-405d-b3cf-de42924b33e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr" Oct 30 21:48:39.519454 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.519434 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8zmx\" (UniqueName: \"kubernetes.io/projected/bbc85c6d-b1d2-405d-b3cf-de42924b33e3-kube-api-access-w8zmx\") pod \"maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr\" (UID: \"bbc85c6d-b1d2-405d-b3cf-de42924b33e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr" Oct 30 21:48:39.526143 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.526118 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11886179-f8d2-4237-a992-8103eed3ad7c" path="/var/lib/kubelet/pods/11886179-f8d2-4237-a992-8103eed3ad7c/volumes" Oct 30 21:48:39.620736 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.620698 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbc85c6d-b1d2-405d-b3cf-de42924b33e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr\" (UID: \"bbc85c6d-b1d2-405d-b3cf-de42924b33e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr" Oct 30 21:48:39.620956 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.620936 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8zmx\" (UniqueName: \"kubernetes.io/projected/bbc85c6d-b1d2-405d-b3cf-de42924b33e3-kube-api-access-w8zmx\") pod \"maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr\" (UID: \"bbc85c6d-b1d2-405d-b3cf-de42924b33e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr" Oct 30 21:48:39.621163 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.621143 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbc85c6d-b1d2-405d-b3cf-de42924b33e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr\" (UID: \"bbc85c6d-b1d2-405d-b3cf-de42924b33e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr" Oct 30 21:48:39.621494 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.621473 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbc85c6d-b1d2-405d-b3cf-de42924b33e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr\" (UID: \"bbc85c6d-b1d2-405d-b3cf-de42924b33e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr" Oct 30 21:48:39.623432 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.623410 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbc85c6d-b1d2-405d-b3cf-de42924b33e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr\" (UID: \"bbc85c6d-b1d2-405d-b3cf-de42924b33e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr" Oct 30 21:48:39.634561 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.634501 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8zmx\" (UniqueName: \"kubernetes.io/projected/bbc85c6d-b1d2-405d-b3cf-de42924b33e3-kube-api-access-w8zmx\") pod \"maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr\" (UID: \"bbc85c6d-b1d2-405d-b3cf-de42924b33e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr" Oct 30 21:48:39.709709 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.709679 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr" Oct 30 21:48:39.914380 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:39.914292 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:40.033065 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:40.033034 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr"] Oct 30 21:48:40.689555 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:40.689462 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:40.796159 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:40.796122 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr" event={"ID":"bbc85c6d-b1d2-405d-b3cf-de42924b33e3","Type":"ContainerStarted","Data":"e4b55302b15a622bf0267e98cb85b3c2ec076083574a03753163048d08e5ee81"} Oct 30 21:48:40.796340 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:40.796183 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr" event={"ID":"bbc85c6d-b1d2-405d-b3cf-de42924b33e3","Type":"ContainerStarted","Data":"752bceccb4771d0ea12b685390f03b99b3101cd53531ef7fd7d58e72a8395203"} Oct 30 21:48:40.815801 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:40.815686 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr" podStartSLOduration=1.8156681799999999 podStartE2EDuration="1.81566818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:40.815087744 +0000 UTC m=+1189.410904431" watchObservedRunningTime="2026-10-30 21:48:40.81566818 +0000 UTC m=+1189.411484841" Oct 30 21:48:41.469430 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:41.469342 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:41.811921 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:41.811886 5801 generic.go:356] "Generic (PLEG): container finished" podID="bbc85c6d-b1d2-405d-b3cf-de42924b33e3" containerID="e4b55302b15a622bf0267e98cb85b3c2ec076083574a03753163048d08e5ee81" exitCode=0 Oct 30 21:48:41.812118 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:41.811978 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr" event={"ID":"bbc85c6d-b1d2-405d-b3cf-de42924b33e3","Type":"ContainerDied","Data":"e4b55302b15a622bf0267e98cb85b3c2ec076083574a03753163048d08e5ee81"} Oct 30 21:48:42.231268 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:42.231183 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:43.126390 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:43.126284 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:43.233247 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:43.233216 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr" Oct 30 21:48:43.414924 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:43.414835 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc85c6d-b1d2-405d-b3cf-de42924b33e3-kube-api-access-w8zmx\" (UniqueName: \"kubernetes.io/projected/bbc85c6d-b1d2-405d-b3cf-de42924b33e3-kube-api-access-w8zmx\") pod \"bbc85c6d-b1d2-405d-b3cf-de42924b33e3\" (UID: \"bbc85c6d-b1d2-405d-b3cf-de42924b33e3\") " Oct 30 21:48:43.416667 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:43.416636 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbc85c6d-b1d2-405d-b3cf-de42924b33e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbc85c6d-b1d2-405d-b3cf-de42924b33e3-maas-api-openshift-ca\") pod \"bbc85c6d-b1d2-405d-b3cf-de42924b33e3\" (UID: \"bbc85c6d-b1d2-405d-b3cf-de42924b33e3\") " Oct 30 21:48:43.417306 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:43.417279 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbc85c6d-b1d2-405d-b3cf-de42924b33e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbc85c6d-b1d2-405d-b3cf-de42924b33e3-maas-api-xks-ca\") pod \"bbc85c6d-b1d2-405d-b3cf-de42924b33e3\" (UID: \"bbc85c6d-b1d2-405d-b3cf-de42924b33e3\") " Oct 30 21:48:43.418123 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:43.418096 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc85c6d-b1d2-405d-b3cf-de42924b33e3-maas-api-openshift-ca" pod "bbc85c6d-b1d2-405d-b3cf-de42924b33e3" (UID: "bbc85c6d-b1d2-405d-b3cf-de42924b33e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:43.418924 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:43.418901 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbc85c6d-b1d2-405d-b3cf-de42924b33e3-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:43.432937 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:43.421387 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc85c6d-b1d2-405d-b3cf-de42924b33e3-kube-api-access-w8zmx" pod "bbc85c6d-b1d2-405d-b3cf-de42924b33e3" (UID: "bbc85c6d-b1d2-405d-b3cf-de42924b33e3"). InnerVolumeSpecName "kube-api-access-w8zmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:43.432937 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:43.421393 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc85c6d-b1d2-405d-b3cf-de42924b33e3-maas-api-xks-ca" pod "bbc85c6d-b1d2-405d-b3cf-de42924b33e3" (UID: "bbc85c6d-b1d2-405d-b3cf-de42924b33e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:43.520922 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:43.520889 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbc85c6d-b1d2-405d-b3cf-de42924b33e3-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:43.520922 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:43.520918 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8zmx\" (UniqueName: \"kubernetes.io/projected/bbc85c6d-b1d2-405d-b3cf-de42924b33e3-kube-api-access-w8zmx\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:43.832925 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:43.832893 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr" Oct 30 21:48:43.832925 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:43.832915 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr" event={"ID":"bbc85c6d-b1d2-405d-b3cf-de42924b33e3","Type":"ContainerDied","Data":"752bceccb4771d0ea12b685390f03b99b3101cd53531ef7fd7d58e72a8395203"} Oct 30 21:48:43.833134 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:43.832966 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="752bceccb4771d0ea12b685390f03b99b3101cd53531ef7fd7d58e72a8395203" Oct 30 21:48:43.933888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:43.933859 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:44.636420 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:44.636381 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:44.801107 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:44.801073 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:44.903628 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:44.903525 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:45.159822 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:45.143988 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb"] Oct 30 21:48:45.159822 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:45.144181 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" podUID="929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" containerName="istio-proxy" containerID="cri-o://be4676eb2e66fbcb2ba91df95e765b219028be4ff010ed01d60ba4e8e95f068d" gracePeriod=30 Oct 30 21:48:45.159822 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:45.158196 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k"] Oct 30 21:48:45.159822 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:45.158409 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" podUID="402d335b-8671-4c9a-957e-be90693d2085" containerName="istio-proxy" containerID="cri-o://c0d0345dfbbf1d70d1bc2773bfa220919de7091a2ed8bfec43baac00d734827f" gracePeriod=30 Oct 30 21:48:45.199910 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:45.199880 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:45.520665 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:45.520572 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:45.858692 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:45.858656 5801 generic.go:356] "Generic (PLEG): container finished" podID="8690c196-570d-4604-bfee-2a5d84997f90" containerID="a2ee38c889b76a672392ba2971ab7582ffd557d721883da3214c4da9555829ce" exitCode=0 Oct 30 21:48:45.859218 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:45.858700 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" event={"ID":"8690c196-570d-4604-bfee-2a5d84997f90","Type":"ContainerDied","Data":"a2ee38c889b76a672392ba2971ab7582ffd557d721883da3214c4da9555829ce"} Oct 30 21:48:45.859293 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:45.859236 5801 scope.go:122] "RemoveContainer" containerID="a2ee38c889b76a672392ba2971ab7582ffd557d721883da3214c4da9555829ce" Oct 30 21:48:45.861930 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:45.861884 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:48:45.862097 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:45.861990 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:48:45.862097 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:45.862079 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:48:46.013531 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:46.013449 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:46.483693 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:46.483655 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:46.872359 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:46.872314 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" event={"ID":"8690c196-570d-4604-bfee-2a5d84997f90","Type":"ContainerStarted","Data":"ac32c88b29e53e1bc4a7b8d4d87a260e8047df0dde8d567a338c6635d17cf8d0"} Oct 30 21:48:47.009786 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:47.003490 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:47.032835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:47.031603 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:48:47.032835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:47.031726 5801 patch_prober.go:29] interesting pod/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 21:48:47.032835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:47.031798 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" podUID="8690c196-570d-4604-bfee-2a5d84997f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 21:48:47.187626 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:47.187543 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:47.457833 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:47.452152 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:48.026517 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:48.026476 5801 patch_prober.go:29] interesting pod/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 21:48:48.026930 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:48.026534 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" podUID="8690c196-570d-4604-bfee-2a5d84997f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 21:48:48.181572 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:48.181535 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:48.878773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:48.878720 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:49.026142 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:49.026104 5801 patch_prober.go:29] interesting pod/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 21:48:49.026329 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:49.026164 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" podUID="8690c196-570d-4604-bfee-2a5d84997f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 21:48:49.488292 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:49.487581 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t"] Oct 30 21:48:49.582777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:49.582059 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:49.684221 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:49.684163 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:49.785386 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:49.785293 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:50.046106 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.038363 5801 patch_prober.go:29] interesting pod/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 21:48:50.046106 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.038430 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" podUID="8690c196-570d-4604-bfee-2a5d84997f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 21:48:50.073838 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.067124 5801 patch_prober.go:29] interesting pod/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:48:50.073838 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.067201 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" podUID="402d335b-8671-4c9a-957e-be90693d2085" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:48:50.476777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.475995 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:48:50.530444 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.530397 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:50.575830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.553166 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-data\") pod \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " Oct 30 21:48:50.575830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.553229 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-workload-socket\") pod \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " Oct 30 21:48:50.575830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.553288 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-envoy\") pod \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " Oct 30 21:48:50.575830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.553390 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-token\" (UniqueName: \"kubernetes.io/projected/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-token\") pod \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " Oct 30 21:48:50.575830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.553551 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-workload-certs\") pod \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " Oct 30 21:48:50.575830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.553586 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-kube-api-access-4nrwz\" (UniqueName: \"kubernetes.io/projected/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-kube-api-access-4nrwz\") pod \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " Oct 30 21:48:50.575830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.553639 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istiod-ca-cert\") pod \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " Oct 30 21:48:50.575830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.553691 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-credential-socket\") pod \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " Oct 30 21:48:50.575830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.553893 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-podinfo\") pod \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\" (UID: \"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf\") " Oct 30 21:48:50.578212 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.578177 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-workload-certs" pod "929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" (UID: "929ae553-a934-4cfb-a8e3-5d6c0dfeccdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:50.579390 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.578889 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-workload-socket" pod "929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" (UID: "929ae553-a934-4cfb-a8e3-5d6c0dfeccdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:50.579390 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.579082 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-data" pod "929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" (UID: "929ae553-a934-4cfb-a8e3-5d6c0dfeccdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:50.580075 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.579944 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-credential-socket" pod "929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" (UID: "929ae553-a934-4cfb-a8e3-5d6c0dfeccdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:50.580075 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.580035 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istiod-ca-cert" pod "929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" (UID: "929ae553-a934-4cfb-a8e3-5d6c0dfeccdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:50.580938 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.580911 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-token" pod "929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" (UID: "929ae553-a934-4cfb-a8e3-5d6c0dfeccdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:50.581702 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.581675 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-envoy" pod "929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" (UID: "929ae553-a934-4cfb-a8e3-5d6c0dfeccdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:50.581860 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.581681 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-podinfo" pod "929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" (UID: "929ae553-a934-4cfb-a8e3-5d6c0dfeccdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:48:50.583396 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.583367 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-kube-api-access-4nrwz" pod "929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" (UID: "929ae553-a934-4cfb-a8e3-5d6c0dfeccdf"). InnerVolumeSpecName "kube-api-access-4nrwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:50.684276 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.684113 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:50.684276 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.684151 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:50.684276 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.684169 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nrwz\" (UniqueName: \"kubernetes.io/projected/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-kube-api-access-4nrwz\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:50.684276 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.684185 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:50.684276 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.684199 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:50.684276 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.684213 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:50.684276 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.684227 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:50.684980 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.684241 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:50.684980 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.684960 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:50.798187 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.798133 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc"] Oct 30 21:48:50.799796 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.799543 5801 cpu_manager.go:414] "containerMap: removing container" podUID="bbc85c6d-b1d2-405d-b3cf-de42924b33e3" containerName="revoke-keys" Oct 30 21:48:50.799942 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.799896 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc85c6d-b1d2-405d-b3cf-de42924b33e3" containerName="revoke-keys" Oct 30 21:48:50.799942 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.799930 5801 cpu_manager.go:414] "containerMap: removing container" podUID="929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" containerName="istio-proxy" Oct 30 21:48:50.799942 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.799940 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" containerName="istio-proxy" Oct 30 21:48:50.800111 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.800000 5801 cpu_manager.go:414] "containerMap: removing container" podUID="929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" containerName="istio-proxy" Oct 30 21:48:50.800111 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.800024 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" containerName="istio-proxy" Oct 30 21:48:50.800300 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.800279 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc85c6d-b1d2-405d-b3cf-de42924b33e3" containerName="revoke-keys" Oct 30 21:48:50.800376 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.800302 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" containerName="istio-proxy" Oct 30 21:48:50.812811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.812785 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc"] Oct 30 21:48:50.813119 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.813032 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc" Oct 30 21:48:50.820690 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.819961 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:48:50.820690 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.820425 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:48:50.823037 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.823017 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:48:50.823288 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.823270 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:48:50.893402 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.889994 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3aef510-0168-418b-96db-214b7da928a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc\" (UID: \"b3aef510-0168-418b-96db-214b7da928a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc" Oct 30 21:48:50.893402 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.890164 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqtsx\" (UniqueName: \"kubernetes.io/projected/b3aef510-0168-418b-96db-214b7da928a9-kube-api-access-lqtsx\") pod \"maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc\" (UID: \"b3aef510-0168-418b-96db-214b7da928a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc" Oct 30 21:48:50.893402 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.890411 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3aef510-0168-418b-96db-214b7da928a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc\" (UID: \"b3aef510-0168-418b-96db-214b7da928a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc" Oct 30 21:48:50.948424 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.948399 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:48:50.996058 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.996017 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqtsx\" (UniqueName: \"kubernetes.io/projected/b3aef510-0168-418b-96db-214b7da928a9-kube-api-access-lqtsx\") pod \"maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc\" (UID: \"b3aef510-0168-418b-96db-214b7da928a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc" Oct 30 21:48:50.996299 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.996276 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3aef510-0168-418b-96db-214b7da928a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc\" (UID: \"b3aef510-0168-418b-96db-214b7da928a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc" Oct 30 21:48:50.996428 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.996413 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3aef510-0168-418b-96db-214b7da928a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc\" (UID: \"b3aef510-0168-418b-96db-214b7da928a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc" Oct 30 21:48:50.998989 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.998968 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3aef510-0168-418b-96db-214b7da928a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc\" (UID: \"b3aef510-0168-418b-96db-214b7da928a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc" Oct 30 21:48:50.999544 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.999519 5801 generic.go:356] "Generic (PLEG): container finished" podID="929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" containerID="be4676eb2e66fbcb2ba91df95e765b219028be4ff010ed01d60ba4e8e95f068d" exitCode=0 Oct 30 21:48:50.999651 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.999572 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" event={"ID":"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf","Type":"ContainerDied","Data":"be4676eb2e66fbcb2ba91df95e765b219028be4ff010ed01d60ba4e8e95f068d"} Oct 30 21:48:50.999651 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.999590 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" Oct 30 21:48:50.999651 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.999641 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb" event={"ID":"929ae553-a934-4cfb-a8e3-5d6c0dfeccdf","Type":"ContainerDied","Data":"a47bff197fe163398037b914aa180772d96aeb774f95e4a35131e19c8279e9a7"} Oct 30 21:48:50.999882 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:50.999673 5801 scope.go:122] "RemoveContainer" containerID="be4676eb2e66fbcb2ba91df95e765b219028be4ff010ed01d60ba4e8e95f068d" Oct 30 21:48:51.001649 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.001622 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3aef510-0168-418b-96db-214b7da928a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc\" (UID: \"b3aef510-0168-418b-96db-214b7da928a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc" Oct 30 21:48:51.002738 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.002716 5801 generic.go:356] "Generic (PLEG): container finished" podID="402d335b-8671-4c9a-957e-be90693d2085" containerID="c0d0345dfbbf1d70d1bc2773bfa220919de7091a2ed8bfec43baac00d734827f" exitCode=0 Oct 30 21:48:51.002873 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.002770 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" Oct 30 21:48:51.003186 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.002974 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" event={"ID":"402d335b-8671-4c9a-957e-be90693d2085","Type":"ContainerDied","Data":"c0d0345dfbbf1d70d1bc2773bfa220919de7091a2ed8bfec43baac00d734827f"} Oct 30 21:48:51.003186 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.003007 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k" event={"ID":"402d335b-8671-4c9a-957e-be90693d2085","Type":"ContainerDied","Data":"69ca82f5bf4ddef2211ed736c343b1d3aa2771c7f002cc17e04311ed032b5540"} Oct 30 21:48:51.011587 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.011571 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqtsx\" (UniqueName: \"kubernetes.io/projected/b3aef510-0168-418b-96db-214b7da928a9-kube-api-access-lqtsx\") pod \"maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc\" (UID: \"b3aef510-0168-418b-96db-214b7da928a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc" Oct 30 21:48:51.017374 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.017355 5801 scope.go:122] "RemoveContainer" containerID="f57ffb3b59ac5ad6024b8b8e95001f7fabea9c4c10c94b3787e929cac75b5b2e" Oct 30 21:48:51.026189 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.026165 5801 patch_prober.go:29] interesting pod/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 21:48:51.026372 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.026349 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" podUID="8690c196-570d-4604-bfee-2a5d84997f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 21:48:51.030212 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.029807 5801 scope.go:122] "RemoveContainer" containerID="be4676eb2e66fbcb2ba91df95e765b219028be4ff010ed01d60ba4e8e95f068d" Oct 30 21:48:51.030389 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:48:51.030342 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4676eb2e66fbcb2ba91df95e765b219028be4ff010ed01d60ba4e8e95f068d\": container with ID starting with be4676eb2e66fbcb2ba91df95e765b219028be4ff010ed01d60ba4e8e95f068d not found: ID does not exist" containerID="be4676eb2e66fbcb2ba91df95e765b219028be4ff010ed01d60ba4e8e95f068d" Oct 30 21:48:51.030951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.030386 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4676eb2e66fbcb2ba91df95e765b219028be4ff010ed01d60ba4e8e95f068d"} err="failed to get container status \"be4676eb2e66fbcb2ba91df95e765b219028be4ff010ed01d60ba4e8e95f068d\": rpc error: code = NotFound desc = could not find container \"be4676eb2e66fbcb2ba91df95e765b219028be4ff010ed01d60ba4e8e95f068d\": container with ID starting with be4676eb2e66fbcb2ba91df95e765b219028be4ff010ed01d60ba4e8e95f068d not found: ID does not exist" Oct 30 21:48:51.030951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.030467 5801 scope.go:122] "RemoveContainer" containerID="f57ffb3b59ac5ad6024b8b8e95001f7fabea9c4c10c94b3787e929cac75b5b2e" Oct 30 21:48:51.034320 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:48:51.033790 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f57ffb3b59ac5ad6024b8b8e95001f7fabea9c4c10c94b3787e929cac75b5b2e\": container with ID starting with f57ffb3b59ac5ad6024b8b8e95001f7fabea9c4c10c94b3787e929cac75b5b2e not found: ID does not exist" containerID="f57ffb3b59ac5ad6024b8b8e95001f7fabea9c4c10c94b3787e929cac75b5b2e" Oct 30 21:48:51.034320 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.033822 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f57ffb3b59ac5ad6024b8b8e95001f7fabea9c4c10c94b3787e929cac75b5b2e"} err="failed to get container status \"f57ffb3b59ac5ad6024b8b8e95001f7fabea9c4c10c94b3787e929cac75b5b2e\": rpc error: code = NotFound desc = could not find container \"f57ffb3b59ac5ad6024b8b8e95001f7fabea9c4c10c94b3787e929cac75b5b2e\": container with ID starting with f57ffb3b59ac5ad6024b8b8e95001f7fabea9c4c10c94b3787e929cac75b5b2e not found: ID does not exist" Oct 30 21:48:51.034320 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.033844 5801 scope.go:122] "RemoveContainer" containerID="c0d0345dfbbf1d70d1bc2773bfa220919de7091a2ed8bfec43baac00d734827f" Oct 30 21:48:51.041052 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.041034 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb"] Oct 30 21:48:51.050073 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.049107 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-qbklb"] Oct 30 21:48:51.088174 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.088110 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:51.095926 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.095887 5801 scope.go:122] "RemoveContainer" containerID="c0d0345dfbbf1d70d1bc2773bfa220919de7091a2ed8bfec43baac00d734827f" Oct 30 21:48:51.096655 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:48:51.096629 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0d0345dfbbf1d70d1bc2773bfa220919de7091a2ed8bfec43baac00d734827f\": container with ID starting with c0d0345dfbbf1d70d1bc2773bfa220919de7091a2ed8bfec43baac00d734827f not found: ID does not exist" containerID="c0d0345dfbbf1d70d1bc2773bfa220919de7091a2ed8bfec43baac00d734827f" Oct 30 21:48:51.096736 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.096666 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0d0345dfbbf1d70d1bc2773bfa220919de7091a2ed8bfec43baac00d734827f"} err="failed to get container status \"c0d0345dfbbf1d70d1bc2773bfa220919de7091a2ed8bfec43baac00d734827f\": rpc error: code = NotFound desc = could not find container \"c0d0345dfbbf1d70d1bc2773bfa220919de7091a2ed8bfec43baac00d734827f\": container with ID starting with c0d0345dfbbf1d70d1bc2773bfa220919de7091a2ed8bfec43baac00d734827f not found: ID does not exist" Oct 30 21:48:51.097103 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.097086 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/402d335b-8671-4c9a-957e-be90693d2085-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/402d335b-8671-4c9a-957e-be90693d2085-istiod-ca-cert\") pod \"402d335b-8671-4c9a-957e-be90693d2085\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " Oct 30 21:48:51.097180 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.097113 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/402d335b-8671-4c9a-957e-be90693d2085-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/402d335b-8671-4c9a-957e-be90693d2085-istio-podinfo\") pod \"402d335b-8671-4c9a-957e-be90693d2085\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " Oct 30 21:48:51.097273 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.097257 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/402d335b-8671-4c9a-957e-be90693d2085-istio-token\" (UniqueName: \"kubernetes.io/projected/402d335b-8671-4c9a-957e-be90693d2085-istio-token\") pod \"402d335b-8671-4c9a-957e-be90693d2085\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " Oct 30 21:48:51.097320 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.097302 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-istio-data\") pod \"402d335b-8671-4c9a-957e-be90693d2085\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " Oct 30 21:48:51.097465 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.097449 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-workload-certs\") pod \"402d335b-8671-4c9a-957e-be90693d2085\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " Oct 30 21:48:51.097501 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.097483 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/402d335b-8671-4c9a-957e-be90693d2085-kube-api-access-lpjsr\" (UniqueName: \"kubernetes.io/projected/402d335b-8671-4c9a-957e-be90693d2085-kube-api-access-lpjsr\") pod \"402d335b-8671-4c9a-957e-be90693d2085\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " Oct 30 21:48:51.097536 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.097523 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-istio-envoy\") pod \"402d335b-8671-4c9a-957e-be90693d2085\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " Oct 30 21:48:51.097676 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.097660 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-credential-socket\") pod \"402d335b-8671-4c9a-957e-be90693d2085\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " Oct 30 21:48:51.097785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.097765 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-workload-socket\") pod \"402d335b-8671-4c9a-957e-be90693d2085\" (UID: \"402d335b-8671-4c9a-957e-be90693d2085\") " Oct 30 21:48:51.097886 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.097787 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-workload-certs" pod "402d335b-8671-4c9a-957e-be90693d2085" (UID: "402d335b-8671-4c9a-957e-be90693d2085"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:51.097984 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.097962 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-istio-data" pod "402d335b-8671-4c9a-957e-be90693d2085" (UID: "402d335b-8671-4c9a-957e-be90693d2085"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:51.098296 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.098270 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/402d335b-8671-4c9a-957e-be90693d2085-istiod-ca-cert" pod "402d335b-8671-4c9a-957e-be90693d2085" (UID: "402d335b-8671-4c9a-957e-be90693d2085"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:51.098431 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.098397 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-workload-socket" pod "402d335b-8671-4c9a-957e-be90693d2085" (UID: "402d335b-8671-4c9a-957e-be90693d2085"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:51.098431 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.098116 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-credential-socket" pod "402d335b-8671-4c9a-957e-be90693d2085" (UID: "402d335b-8671-4c9a-957e-be90693d2085"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:51.098967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.098947 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:51.098967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.098967 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:51.099130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.098978 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/402d335b-8671-4c9a-957e-be90693d2085-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:51.099130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.098986 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:51.099130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.099011 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:51.100080 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.100053 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402d335b-8671-4c9a-957e-be90693d2085-istio-token" pod "402d335b-8671-4c9a-957e-be90693d2085" (UID: "402d335b-8671-4c9a-957e-be90693d2085"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:51.100962 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.100923 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/402d335b-8671-4c9a-957e-be90693d2085-istio-podinfo" pod "402d335b-8671-4c9a-957e-be90693d2085" (UID: "402d335b-8671-4c9a-957e-be90693d2085"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:48:51.101089 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.101074 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-istio-envoy" pod "402d335b-8671-4c9a-957e-be90693d2085" (UID: "402d335b-8671-4c9a-957e-be90693d2085"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:51.101739 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.101718 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402d335b-8671-4c9a-957e-be90693d2085-kube-api-access-lpjsr" pod "402d335b-8671-4c9a-957e-be90693d2085" (UID: "402d335b-8671-4c9a-957e-be90693d2085"). InnerVolumeSpecName "kube-api-access-lpjsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:51.139232 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.139211 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc" Oct 30 21:48:51.200747 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.200717 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/402d335b-8671-4c9a-957e-be90693d2085-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:51.200895 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.200763 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpjsr\" (UniqueName: \"kubernetes.io/projected/402d335b-8671-4c9a-957e-be90693d2085-kube-api-access-lpjsr\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:51.200895 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.200781 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/402d335b-8671-4c9a-957e-be90693d2085-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:51.200895 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.200790 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/402d335b-8671-4c9a-957e-be90693d2085-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:51.262834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.259328 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:51.420780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.417364 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k"] Oct 30 21:48:51.440781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.435309 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7386906d-openshift-default-77bf4c6869-rz67k"] Oct 30 21:48:51.538668 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.538633 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="402d335b-8671-4c9a-957e-be90693d2085" path="/var/lib/kubelet/pods/402d335b-8671-4c9a-957e-be90693d2085/volumes" Oct 30 21:48:51.539434 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.539411 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" path="/var/lib/kubelet/pods/929ae553-a934-4cfb-a8e3-5d6c0dfeccdf/volumes" Oct 30 21:48:51.712713 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:51.712640 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc"] Oct 30 21:48:51.722702 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:48:51.722670 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3aef510_0168_418b_96db_214b7da928a9.slice/crio-765b453a8af01080ea11fb8a0e4ac0b5855d140f47cf69bd61d7d0517d141484 WatchSource:0}: Error finding container 765b453a8af01080ea11fb8a0e4ac0b5855d140f47cf69bd61d7d0517d141484: Status 404 returned error can't find the container with id 765b453a8af01080ea11fb8a0e4ac0b5855d140f47cf69bd61d7d0517d141484 Oct 30 21:48:52.014439 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:52.014405 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc" event={"ID":"b3aef510-0168-418b-96db-214b7da928a9","Type":"ContainerStarted","Data":"765b453a8af01080ea11fb8a0e4ac0b5855d140f47cf69bd61d7d0517d141484"} Oct 30 21:48:52.025901 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:52.025869 5801 patch_prober.go:29] interesting pod/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 21:48:52.026055 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:52.025939 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" podUID="8690c196-570d-4604-bfee-2a5d84997f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 21:48:52.431022 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:52.430980 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:52.733870 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:52.726114 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:52.775780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:52.773320 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j"] Oct 30 21:48:52.775780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:52.774585 5801 cpu_manager.go:414] "containerMap: removing container" podUID="402d335b-8671-4c9a-957e-be90693d2085" containerName="istio-proxy" Oct 30 21:48:52.775780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:52.774607 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="402d335b-8671-4c9a-957e-be90693d2085" containerName="istio-proxy" Oct 30 21:48:52.775780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:52.774830 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="929ae553-a934-4cfb-a8e3-5d6c0dfeccdf" containerName="istio-proxy" Oct 30 21:48:52.775780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:52.774855 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="402d335b-8671-4c9a-957e-be90693d2085" containerName="istio-proxy" Oct 30 21:48:52.779799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:52.779774 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j" Oct 30 21:48:52.807811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:52.807551 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37caf73-c0d3-4433-8877-ee88415dd6c6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j\" (UID: \"d37caf73-c0d3-4433-8877-ee88415dd6c6\") " pod="openshift-ingress/payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j" Oct 30 21:48:52.807811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:52.807744 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srnqd\" (UniqueName: \"kubernetes.io/projected/d37caf73-c0d3-4433-8877-ee88415dd6c6-kube-api-access-srnqd\") pod \"payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j\" (UID: \"d37caf73-c0d3-4433-8877-ee88415dd6c6\") " pod="openshift-ingress/payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j" Oct 30 21:48:52.814301 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:52.808226 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f8144d\"" Oct 30 21:48:52.820771 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:52.820728 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j"] Oct 30 21:48:52.829152 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:52.829113 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl"] Oct 30 21:48:52.836437 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:52.835929 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl" Oct 30 21:48:52.900778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:52.895127 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f8144d-dockercfg-7dck6\"" Oct 30 21:48:52.933865 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:52.917619 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37caf73-c0d3-4433-8877-ee88415dd6c6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j\" (UID: \"d37caf73-c0d3-4433-8877-ee88415dd6c6\") " pod="openshift-ingress/payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j" Oct 30 21:48:52.953773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:52.934507 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srnqd\" (UniqueName: \"kubernetes.io/projected/d37caf73-c0d3-4433-8877-ee88415dd6c6-kube-api-access-srnqd\") pod \"payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j\" (UID: \"d37caf73-c0d3-4433-8877-ee88415dd6c6\") " pod="openshift-ingress/payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j" Oct 30 21:48:52.953773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:52.947563 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:53.072060 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:53.065081 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37caf73-c0d3-4433-8877-ee88415dd6c6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j\" (UID: \"d37caf73-c0d3-4433-8877-ee88415dd6c6\") " pod="openshift-ingress/payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j" Oct 30 21:48:53.074919 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:53.074889 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srnqd\" (UniqueName: \"kubernetes.io/projected/d37caf73-c0d3-4433-8877-ee88415dd6c6-kube-api-access-srnqd\") pod \"payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j\" (UID: \"d37caf73-c0d3-4433-8877-ee88415dd6c6\") " pod="openshift-ingress/payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j" Oct 30 21:48:53.078702 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:53.077997 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc" event={"ID":"b3aef510-0168-418b-96db-214b7da928a9","Type":"ContainerStarted","Data":"2de5dc4f44715b56cdda960ee1f7dc791d742a91602124b61f3c65a78dd223b0"} Oct 30 21:48:53.078702 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:53.078062 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl"] Oct 30 21:48:53.109709 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:53.109674 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zdkb\" (UniqueName: \"kubernetes.io/projected/8be262d1-439b-4a2e-8658-e3ae067edc61-kube-api-access-4zdkb\") pod \"payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl\" (UID: \"8be262d1-439b-4a2e-8658-e3ae067edc61\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl" Oct 30 21:48:53.109981 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:53.109964 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be262d1-439b-4a2e-8658-e3ae067edc61-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl\" (UID: \"8be262d1-439b-4a2e-8658-e3ae067edc61\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl" Oct 30 21:48:53.126056 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:53.125897 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j" Oct 30 21:48:53.131467 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:53.131425 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc" podStartSLOduration=3.13141333 podStartE2EDuration="3.13141333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:53.126611855 +0000 UTC m=+1201.722428527" watchObservedRunningTime="2026-10-30 21:48:53.13141333 +0000 UTC m=+1201.727229994" Oct 30 21:48:53.189099 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:53.189068 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:48:53.191243 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:53.191218 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:48:53.192712 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:53.192690 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:48:53.211850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:53.211819 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be262d1-439b-4a2e-8658-e3ae067edc61-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl\" (UID: \"8be262d1-439b-4a2e-8658-e3ae067edc61\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl" Oct 30 21:48:53.212042 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:53.212027 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zdkb\" (UniqueName: \"kubernetes.io/projected/8be262d1-439b-4a2e-8658-e3ae067edc61-kube-api-access-4zdkb\") pod \"payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl\" (UID: \"8be262d1-439b-4a2e-8658-e3ae067edc61\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl" Oct 30 21:48:53.214981 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:53.214096 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be262d1-439b-4a2e-8658-e3ae067edc61-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl\" (UID: \"8be262d1-439b-4a2e-8658-e3ae067edc61\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl" Oct 30 21:48:53.291508 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:53.291473 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zdkb\" (UniqueName: \"kubernetes.io/projected/8be262d1-439b-4a2e-8658-e3ae067edc61-kube-api-access-4zdkb\") pod \"payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl\" (UID: \"8be262d1-439b-4a2e-8658-e3ae067edc61\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl" Oct 30 21:48:53.500132 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:53.476104 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl" Oct 30 21:48:53.516925 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:48:53.508144 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37caf73_c0d3_4433_8877_ee88415dd6c6.slice/crio-a2b98641a775fa5ffa0006c776cc042ee1874ab05e8cb478b889982ee480962f WatchSource:0}: Error finding container a2b98641a775fa5ffa0006c776cc042ee1874ab05e8cb478b889982ee480962f: Status 404 returned error can't find the container with id a2b98641a775fa5ffa0006c776cc042ee1874ab05e8cb478b889982ee480962f Oct 30 21:48:53.594137 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:53.589266 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j"] Oct 30 21:48:53.872417 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:48:53.872370 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be262d1_439b_4a2e_8658_e3ae067edc61.slice/crio-54876a52528427e07524c38164deb899c7898f66b1165c5da46ac9a1050659db WatchSource:0}: Error finding container 54876a52528427e07524c38164deb899c7898f66b1165c5da46ac9a1050659db: Status 404 returned error can't find the container with id 54876a52528427e07524c38164deb899c7898f66b1165c5da46ac9a1050659db Oct 30 21:48:53.873072 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:53.873048 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl"] Oct 30 21:48:54.012483 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:54.012444 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:54.089317 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:54.089284 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl" event={"ID":"8be262d1-439b-4a2e-8658-e3ae067edc61","Type":"ContainerStarted","Data":"f35b82c230a040af87fdcc775e5677097ef8e85f67f31d908db994890601a9e1"} Oct 30 21:48:54.089509 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:54.089337 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl" event={"ID":"8be262d1-439b-4a2e-8658-e3ae067edc61","Type":"ContainerStarted","Data":"54876a52528427e07524c38164deb899c7898f66b1165c5da46ac9a1050659db"} Oct 30 21:48:54.089570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:54.089508 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl" Oct 30 21:48:54.092669 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:54.091200 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j" event={"ID":"d37caf73-c0d3-4433-8877-ee88415dd6c6","Type":"ContainerStarted","Data":"66618bf5f4000e93e6c657e7a3ffe6b8308f2a29aa99e02647ea636e195d07db"} Oct 30 21:48:54.092669 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:54.091260 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j" event={"ID":"d37caf73-c0d3-4433-8877-ee88415dd6c6","Type":"ContainerStarted","Data":"a2b98641a775fa5ffa0006c776cc042ee1874ab05e8cb478b889982ee480962f"} Oct 30 21:48:54.092669 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:54.091290 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j" Oct 30 21:48:54.093268 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:54.093239 5801 generic.go:356] "Generic (PLEG): container finished" podID="b3aef510-0168-418b-96db-214b7da928a9" containerID="2de5dc4f44715b56cdda960ee1f7dc791d742a91602124b61f3c65a78dd223b0" exitCode=0 Oct 30 21:48:54.093268 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:54.093265 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc" event={"ID":"b3aef510-0168-418b-96db-214b7da928a9","Type":"ContainerDied","Data":"2de5dc4f44715b56cdda960ee1f7dc791d742a91602124b61f3c65a78dd223b0"} Oct 30 21:48:54.112874 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:54.112799 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl" podStartSLOduration=2.112781724 podStartE2EDuration="2.112781724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:54.108446372 +0000 UTC m=+1202.704263045" watchObservedRunningTime="2026-10-30 21:48:54.112781724 +0000 UTC m=+1202.708598385" Oct 30 21:48:54.143009 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:54.142905 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j" podStartSLOduration=2.142889927 podStartE2EDuration="2.142889927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:54.139194509 +0000 UTC m=+1202.735011169" watchObservedRunningTime="2026-10-30 21:48:54.142889927 +0000 UTC m=+1202.738706589" Oct 30 21:48:54.488584 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:54.488479 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t" podUID="6bd5b368-4306-4836-b13a-e3ada8861d3b" containerName="payload-processing" containerID="cri-o://77fa14775c0f12c987c172f86a847c9f20751a56f438dd59a9b64dbebb34e500" gracePeriod=25 Oct 30 21:48:54.913641 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:54.913604 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t" Oct 30 21:48:55.027040 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.026987 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bd5b368-4306-4836-b13a-e3ada8861d3b-kube-api-access-jnffb\" (UniqueName: \"kubernetes.io/projected/6bd5b368-4306-4836-b13a-e3ada8861d3b-kube-api-access-jnffb\") pod \"6bd5b368-4306-4836-b13a-e3ada8861d3b\" (UID: \"6bd5b368-4306-4836-b13a-e3ada8861d3b\") " Oct 30 21:48:55.027511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.027478 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bd5b368-4306-4836-b13a-e3ada8861d3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bd5b368-4306-4836-b13a-e3ada8861d3b-plugins-config-volume\") pod \"6bd5b368-4306-4836-b13a-e3ada8861d3b\" (UID: \"6bd5b368-4306-4836-b13a-e3ada8861d3b\") " Oct 30 21:48:55.028794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.028745 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd5b368-4306-4836-b13a-e3ada8861d3b-plugins-config-volume" pod "6bd5b368-4306-4836-b13a-e3ada8861d3b" (UID: "6bd5b368-4306-4836-b13a-e3ada8861d3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:55.098859 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.079936 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd5b368-4306-4836-b13a-e3ada8861d3b-kube-api-access-jnffb" pod "6bd5b368-4306-4836-b13a-e3ada8861d3b" (UID: "6bd5b368-4306-4836-b13a-e3ada8861d3b"). InnerVolumeSpecName "kube-api-access-jnffb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:55.119553 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.119512 5801 generic.go:356] "Generic (PLEG): container finished" podID="6bd5b368-4306-4836-b13a-e3ada8861d3b" containerID="77fa14775c0f12c987c172f86a847c9f20751a56f438dd59a9b64dbebb34e500" exitCode=0 Oct 30 21:48:55.119780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.119619 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t" Oct 30 21:48:55.120676 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.120644 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t" event={"ID":"6bd5b368-4306-4836-b13a-e3ada8861d3b","Type":"ContainerDied","Data":"77fa14775c0f12c987c172f86a847c9f20751a56f438dd59a9b64dbebb34e500"} Oct 30 21:48:55.120815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.120709 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t" event={"ID":"6bd5b368-4306-4836-b13a-e3ada8861d3b","Type":"ContainerDied","Data":"f38b1ed352510ff6f08ffc976167801f54427f445ff2bfe8b9586cbb9f49602f"} Oct 30 21:48:55.120815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.120743 5801 scope.go:122] "RemoveContainer" containerID="77fa14775c0f12c987c172f86a847c9f20751a56f438dd59a9b64dbebb34e500" Oct 30 21:48:55.133656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.133634 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnffb\" (UniqueName: \"kubernetes.io/projected/6bd5b368-4306-4836-b13a-e3ada8861d3b-kube-api-access-jnffb\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:55.133656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.133658 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bd5b368-4306-4836-b13a-e3ada8861d3b-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:55.144447 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.144426 5801 scope.go:122] "RemoveContainer" containerID="77fa14775c0f12c987c172f86a847c9f20751a56f438dd59a9b64dbebb34e500" Oct 30 21:48:55.144809 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:48:55.144745 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77fa14775c0f12c987c172f86a847c9f20751a56f438dd59a9b64dbebb34e500\": container with ID starting with 77fa14775c0f12c987c172f86a847c9f20751a56f438dd59a9b64dbebb34e500 not found: ID does not exist" containerID="77fa14775c0f12c987c172f86a847c9f20751a56f438dd59a9b64dbebb34e500" Oct 30 21:48:55.144868 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.144820 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77fa14775c0f12c987c172f86a847c9f20751a56f438dd59a9b64dbebb34e500"} err="failed to get container status \"77fa14775c0f12c987c172f86a847c9f20751a56f438dd59a9b64dbebb34e500\": rpc error: code = NotFound desc = could not find container \"77fa14775c0f12c987c172f86a847c9f20751a56f438dd59a9b64dbebb34e500\": container with ID starting with 77fa14775c0f12c987c172f86a847c9f20751a56f438dd59a9b64dbebb34e500 not found: ID does not exist" Oct 30 21:48:55.159986 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.159952 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t"] Oct 30 21:48:55.164088 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.164055 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd3c4c-75b9dd7bc5-n5w2t"] Oct 30 21:48:55.195720 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.195688 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:55.339078 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.339049 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc" Oct 30 21:48:55.452469 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.452382 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3aef510-0168-418b-96db-214b7da928a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3aef510-0168-418b-96db-214b7da928a9-maas-api-xks-ca\") pod \"b3aef510-0168-418b-96db-214b7da928a9\" (UID: \"b3aef510-0168-418b-96db-214b7da928a9\") " Oct 30 21:48:55.452625 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.452544 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3aef510-0168-418b-96db-214b7da928a9-kube-api-access-lqtsx\" (UniqueName: \"kubernetes.io/projected/b3aef510-0168-418b-96db-214b7da928a9-kube-api-access-lqtsx\") pod \"b3aef510-0168-418b-96db-214b7da928a9\" (UID: \"b3aef510-0168-418b-96db-214b7da928a9\") " Oct 30 21:48:55.452930 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.452903 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3aef510-0168-418b-96db-214b7da928a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3aef510-0168-418b-96db-214b7da928a9-maas-api-openshift-ca\") pod \"b3aef510-0168-418b-96db-214b7da928a9\" (UID: \"b3aef510-0168-418b-96db-214b7da928a9\") " Oct 30 21:48:55.453564 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.453540 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3aef510-0168-418b-96db-214b7da928a9-maas-api-openshift-ca" pod "b3aef510-0168-418b-96db-214b7da928a9" (UID: "b3aef510-0168-418b-96db-214b7da928a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:55.453810 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.453740 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3aef510-0168-418b-96db-214b7da928a9-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:55.454987 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.454963 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3aef510-0168-418b-96db-214b7da928a9-maas-api-xks-ca" pod "b3aef510-0168-418b-96db-214b7da928a9" (UID: "b3aef510-0168-418b-96db-214b7da928a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:55.455541 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.455521 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3aef510-0168-418b-96db-214b7da928a9-kube-api-access-lqtsx" pod "b3aef510-0168-418b-96db-214b7da928a9" (UID: "b3aef510-0168-418b-96db-214b7da928a9"). InnerVolumeSpecName "kube-api-access-lqtsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:55.466955 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.466931 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw"] Oct 30 21:48:55.487679 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.487656 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n"] Oct 30 21:48:55.497207 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.497184 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2"] Oct 30 21:48:55.528259 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.528222 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bd5b368-4306-4836-b13a-e3ada8861d3b" path="/var/lib/kubelet/pods/6bd5b368-4306-4836-b13a-e3ada8861d3b/volumes" Oct 30 21:48:55.555139 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.555111 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3aef510-0168-418b-96db-214b7da928a9-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:55.555237 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.555139 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqtsx\" (UniqueName: \"kubernetes.io/projected/b3aef510-0168-418b-96db-214b7da928a9-kube-api-access-lqtsx\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:55.612879 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:55.611117 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:56.134549 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.134521 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc" Oct 30 21:48:56.135006 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.134520 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc" event={"ID":"b3aef510-0168-418b-96db-214b7da928a9","Type":"ContainerDied","Data":"765b453a8af01080ea11fb8a0e4ac0b5855d140f47cf69bd61d7d0517d141484"} Oct 30 21:48:56.135006 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.134653 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="765b453a8af01080ea11fb8a0e4ac0b5855d140f47cf69bd61d7d0517d141484" Oct 30 21:48:56.176866 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.176745 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:56.267566 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.267526 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t"] Oct 30 21:48:56.268445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.268419 5801 cpu_manager.go:414] "containerMap: removing container" podUID="b3aef510-0168-418b-96db-214b7da928a9" containerName="revoke-keys" Oct 30 21:48:56.268445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.268444 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3aef510-0168-418b-96db-214b7da928a9" containerName="revoke-keys" Oct 30 21:48:56.268683 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.268459 5801 cpu_manager.go:414] "containerMap: removing container" podUID="6bd5b368-4306-4836-b13a-e3ada8861d3b" containerName="payload-processing" Oct 30 21:48:56.268683 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.268466 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bd5b368-4306-4836-b13a-e3ada8861d3b" containerName="payload-processing" Oct 30 21:48:56.268877 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.268713 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3aef510-0168-418b-96db-214b7da928a9" containerName="revoke-keys" Oct 30 21:48:56.268877 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.268730 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bd5b368-4306-4836-b13a-e3ada8861d3b" containerName="payload-processing" Oct 30 21:48:56.272803 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.272783 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t" Oct 30 21:48:56.278898 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.278871 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:48:56.279101 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.279077 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:48:56.281330 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.281301 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:48:56.281435 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.281374 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:48:56.283601 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.283122 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t"] Oct 30 21:48:56.390290 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.390205 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3ef08c1-b100-409a-ab98-f36965c0c826-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t\" (UID: \"b3ef08c1-b100-409a-ab98-f36965c0c826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t" Oct 30 21:48:56.390290 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.390245 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3ef08c1-b100-409a-ab98-f36965c0c826-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t\" (UID: \"b3ef08c1-b100-409a-ab98-f36965c0c826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t" Oct 30 21:48:56.390478 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.390367 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhpl9\" (UniqueName: \"kubernetes.io/projected/b3ef08c1-b100-409a-ab98-f36965c0c826-kube-api-access-vhpl9\") pod \"maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t\" (UID: \"b3ef08c1-b100-409a-ab98-f36965c0c826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t" Oct 30 21:48:56.492382 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.492347 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3ef08c1-b100-409a-ab98-f36965c0c826-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t\" (UID: \"b3ef08c1-b100-409a-ab98-f36965c0c826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t" Oct 30 21:48:56.492382 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.492382 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3ef08c1-b100-409a-ab98-f36965c0c826-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t\" (UID: \"b3ef08c1-b100-409a-ab98-f36965c0c826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t" Oct 30 21:48:56.492590 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.492409 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhpl9\" (UniqueName: \"kubernetes.io/projected/b3ef08c1-b100-409a-ab98-f36965c0c826-kube-api-access-vhpl9\") pod \"maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t\" (UID: \"b3ef08c1-b100-409a-ab98-f36965c0c826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t" Oct 30 21:48:56.493015 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.492987 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3ef08c1-b100-409a-ab98-f36965c0c826-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t\" (UID: \"b3ef08c1-b100-409a-ab98-f36965c0c826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t" Oct 30 21:48:56.494939 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.494915 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3ef08c1-b100-409a-ab98-f36965c0c826-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t\" (UID: \"b3ef08c1-b100-409a-ab98-f36965c0c826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t" Oct 30 21:48:56.509728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.509700 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhpl9\" (UniqueName: \"kubernetes.io/projected/b3ef08c1-b100-409a-ab98-f36965c0c826-kube-api-access-vhpl9\") pod \"maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t\" (UID: \"b3ef08c1-b100-409a-ab98-f36965c0c826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t" Oct 30 21:48:56.592746 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.592713 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t" Oct 30 21:48:56.880708 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.880684 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t"] Oct 30 21:48:56.883872 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:48:56.883846 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ef08c1_b100_409a_ab98_f36965c0c826.slice/crio-c827757699d6641f52a22227dfb0121b89464e908ac27a14b3539ce2cfc8d5fc WatchSource:0}: Error finding container c827757699d6641f52a22227dfb0121b89464e908ac27a14b3539ce2cfc8d5fc: Status 404 returned error can't find the container with id c827757699d6641f52a22227dfb0121b89464e908ac27a14b3539ce2cfc8d5fc Oct 30 21:48:56.889338 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:56.889311 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:57.145859 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:57.145820 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t" event={"ID":"b3ef08c1-b100-409a-ab98-f36965c0c826","Type":"ContainerStarted","Data":"c827757699d6641f52a22227dfb0121b89464e908ac27a14b3539ce2cfc8d5fc"} Oct 30 21:48:57.390043 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:57.390002 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:57.415709 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:57.415668 5801 scope.go:122] "RemoveContainer" containerID="099312abba90ca5fe5e9e8299dd92bdca25be2cca3be07fd6cd76ed1dd3c22ed" Oct 30 21:48:57.435093 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:57.435070 5801 scope.go:122] "RemoveContainer" containerID="b1eef8fb3557ad3fafa3e7cc01e7d49e006ce31f646d412db1eb642db4732d3e" Oct 30 21:48:57.449482 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:57.449446 5801 scope.go:122] "RemoveContainer" containerID="deda7dde7953fdfda55687a3c9e74f6054ee46197e740a0d2b8ec49d2a000472" Oct 30 21:48:57.467876 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:57.467741 5801 scope.go:122] "RemoveContainer" containerID="bd409099995303af1e2c10d8d466ef003af51aba2043407286b44875dbcadae8" Oct 30 21:48:57.493092 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:57.487587 5801 scope.go:122] "RemoveContainer" containerID="e441d15a996b438ec80296ecf826dd0ece631e687e45c22b53048ca6bcea0e55" Oct 30 21:48:57.520414 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:57.520380 5801 scope.go:122] "RemoveContainer" containerID="f14edb308a8ac4b2316f6d76d961eb921e7913510de75ddb0e26b7756d12895c" Oct 30 21:48:57.531843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:57.531822 5801 scope.go:122] "RemoveContainer" containerID="17984719d8c6570ae4bb71ffc0e72ab37d3b640a36ffdf9876c9dc9fdce02a42" Oct 30 21:48:57.543147 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:57.543121 5801 scope.go:122] "RemoveContainer" containerID="eda360ac88b172b2e2c99400252e3c3437360a680905ebb9656a76b083537a77" Oct 30 21:48:57.556430 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:57.556409 5801 scope.go:122] "RemoveContainer" containerID="8c8b4996f9874481677f82b29c48c908bec7e43188cb4f361b4339ccc4dbdebe" Oct 30 21:48:57.567163 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:57.567112 5801 scope.go:122] "RemoveContainer" containerID="b813992fa60129a14f2e0020788330e386868677809b4153e671d434a1b7e9bb" Oct 30 21:48:57.577169 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:57.577151 5801 scope.go:122] "RemoveContainer" containerID="d36dad163ff5cafe8843435eb296643c38324b5268d76b0aa884637759bf0b3f" Oct 30 21:48:58.001420 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:58.001335 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:58.105884 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:58.105853 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm"] Oct 30 21:48:58.110637 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:58.110614 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm" Oct 30 21:48:58.123453 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:58.123427 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm"] Oct 30 21:48:58.157694 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:58.157668 5801 generic.go:356] "Generic (PLEG): container finished" podID="b3ef08c1-b100-409a-ab98-f36965c0c826" containerID="667bbf2a2731b80db9bb3e3480fadf7a06c17cef395f31c259f0b3d83389c849" exitCode=0 Oct 30 21:48:58.158130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:58.158047 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t" event={"ID":"b3ef08c1-b100-409a-ab98-f36965c0c826","Type":"ContainerDied","Data":"667bbf2a2731b80db9bb3e3480fadf7a06c17cef395f31c259f0b3d83389c849"} Oct 30 21:48:58.224193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:58.224161 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:58.233125 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:58.233101 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0c8943-d0ef-4c44-8757-e25e5c8b514e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm\" (UID: \"7e0c8943-d0ef-4c44-8757-e25e5c8b514e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm" Oct 30 21:48:58.233338 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:58.233323 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0c8943-d0ef-4c44-8757-e25e5c8b514e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm\" (UID: \"7e0c8943-d0ef-4c44-8757-e25e5c8b514e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm" Oct 30 21:48:58.233445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:58.233425 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t52tr\" (UniqueName: \"kubernetes.io/projected/7e0c8943-d0ef-4c44-8757-e25e5c8b514e-kube-api-access-t52tr\") pod \"maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm\" (UID: \"7e0c8943-d0ef-4c44-8757-e25e5c8b514e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm" Oct 30 21:48:58.334918 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:58.334883 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t52tr\" (UniqueName: \"kubernetes.io/projected/7e0c8943-d0ef-4c44-8757-e25e5c8b514e-kube-api-access-t52tr\") pod \"maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm\" (UID: \"7e0c8943-d0ef-4c44-8757-e25e5c8b514e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm" Oct 30 21:48:58.335089 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:58.335032 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0c8943-d0ef-4c44-8757-e25e5c8b514e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm\" (UID: \"7e0c8943-d0ef-4c44-8757-e25e5c8b514e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm" Oct 30 21:48:58.335308 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:58.335290 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0c8943-d0ef-4c44-8757-e25e5c8b514e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm\" (UID: \"7e0c8943-d0ef-4c44-8757-e25e5c8b514e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm" Oct 30 21:48:58.335652 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:58.335632 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0c8943-d0ef-4c44-8757-e25e5c8b514e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm\" (UID: \"7e0c8943-d0ef-4c44-8757-e25e5c8b514e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm" Oct 30 21:48:58.337828 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:58.337798 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0c8943-d0ef-4c44-8757-e25e5c8b514e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm\" (UID: \"7e0c8943-d0ef-4c44-8757-e25e5c8b514e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm" Oct 30 21:48:58.347109 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:58.347073 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t52tr\" (UniqueName: \"kubernetes.io/projected/7e0c8943-d0ef-4c44-8757-e25e5c8b514e-kube-api-access-t52tr\") pod \"maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm\" (UID: \"7e0c8943-d0ef-4c44-8757-e25e5c8b514e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm" Oct 30 21:48:58.429770 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:58.429728 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm" Oct 30 21:48:58.723625 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:48:58.723585 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0c8943_d0ef_4c44_8757_e25e5c8b514e.slice/crio-29eaeb88c0124899d5e58dab83d935335ba9bdabe70210e2ed73acb5207449b7 WatchSource:0}: Error finding container 29eaeb88c0124899d5e58dab83d935335ba9bdabe70210e2ed73acb5207449b7: Status 404 returned error can't find the container with id 29eaeb88c0124899d5e58dab83d935335ba9bdabe70210e2ed73acb5207449b7 Oct 30 21:48:58.724792 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:58.724764 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm"] Oct 30 21:48:59.148987 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:59.148949 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:59.167851 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:59.167810 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm" event={"ID":"7e0c8943-d0ef-4c44-8757-e25e5c8b514e","Type":"ContainerStarted","Data":"7d83534b6450113b13484172576d7c633305e6ac9acd7ca6b2a8491a8e85f5b6"} Oct 30 21:48:59.168222 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:59.167908 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm" event={"ID":"7e0c8943-d0ef-4c44-8757-e25e5c8b514e","Type":"ContainerStarted","Data":"29eaeb88c0124899d5e58dab83d935335ba9bdabe70210e2ed73acb5207449b7"} Oct 30 21:48:59.202673 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:59.193146 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm" podStartSLOduration=1.193125313 podStartE2EDuration="1.193125313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:59.186264048 +0000 UTC m=+1207.782080729" watchObservedRunningTime="2026-10-30 21:48:59.193125313 +0000 UTC m=+1207.788941974" Oct 30 21:48:59.406508 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:59.406481 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t" Oct 30 21:48:59.559284 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:59.559252 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ef08c1-b100-409a-ab98-f36965c0c826-kube-api-access-vhpl9\" (UniqueName: \"kubernetes.io/projected/b3ef08c1-b100-409a-ab98-f36965c0c826-kube-api-access-vhpl9\") pod \"b3ef08c1-b100-409a-ab98-f36965c0c826\" (UID: \"b3ef08c1-b100-409a-ab98-f36965c0c826\") " Oct 30 21:48:59.559453 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:59.559426 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3ef08c1-b100-409a-ab98-f36965c0c826-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3ef08c1-b100-409a-ab98-f36965c0c826-maas-api-openshift-ca\") pod \"b3ef08c1-b100-409a-ab98-f36965c0c826\" (UID: \"b3ef08c1-b100-409a-ab98-f36965c0c826\") " Oct 30 21:48:59.559521 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:59.559507 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3ef08c1-b100-409a-ab98-f36965c0c826-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3ef08c1-b100-409a-ab98-f36965c0c826-maas-api-xks-ca\") pod \"b3ef08c1-b100-409a-ab98-f36965c0c826\" (UID: \"b3ef08c1-b100-409a-ab98-f36965c0c826\") " Oct 30 21:48:59.559816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:59.559743 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ef08c1-b100-409a-ab98-f36965c0c826-maas-api-openshift-ca" pod "b3ef08c1-b100-409a-ab98-f36965c0c826" (UID: "b3ef08c1-b100-409a-ab98-f36965c0c826"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:59.560402 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:59.560378 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3ef08c1-b100-409a-ab98-f36965c0c826-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:59.561548 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:59.561515 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ef08c1-b100-409a-ab98-f36965c0c826-kube-api-access-vhpl9" pod "b3ef08c1-b100-409a-ab98-f36965c0c826" (UID: "b3ef08c1-b100-409a-ab98-f36965c0c826"). InnerVolumeSpecName "kube-api-access-vhpl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:59.561638 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:59.561551 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3ef08c1-b100-409a-ab98-f36965c0c826-maas-api-xks-ca" pod "b3ef08c1-b100-409a-ab98-f36965c0c826" (UID: "b3ef08c1-b100-409a-ab98-f36965c0c826"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:59.662947 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:59.662920 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3ef08c1-b100-409a-ab98-f36965c0c826-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:59.662947 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:59.662950 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhpl9\" (UniqueName: \"kubernetes.io/projected/b3ef08c1-b100-409a-ab98-f36965c0c826-kube-api-access-vhpl9\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:59.845121 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:48:59.845017 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:00.177798 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:00.177665 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t" Oct 30 21:49:00.177798 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:00.177671 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t" event={"ID":"b3ef08c1-b100-409a-ab98-f36965c0c826","Type":"ContainerDied","Data":"c827757699d6641f52a22227dfb0121b89464e908ac27a14b3539ce2cfc8d5fc"} Oct 30 21:49:00.178287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:00.177846 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c827757699d6641f52a22227dfb0121b89464e908ac27a14b3539ce2cfc8d5fc" Oct 30 21:49:00.179506 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:00.179482 5801 generic.go:356] "Generic (PLEG): container finished" podID="7e0c8943-d0ef-4c44-8757-e25e5c8b514e" containerID="7d83534b6450113b13484172576d7c633305e6ac9acd7ca6b2a8491a8e85f5b6" exitCode=0 Oct 30 21:49:00.179625 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:00.179517 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm" event={"ID":"7e0c8943-d0ef-4c44-8757-e25e5c8b514e","Type":"ContainerDied","Data":"7d83534b6450113b13484172576d7c633305e6ac9acd7ca6b2a8491a8e85f5b6"} Oct 30 21:49:00.281560 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:49:00.281524 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:49:00.642286 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:00.639041 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:00.935394 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:00.935304 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79"] Oct 30 21:49:01.010171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:01.010140 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:01.110364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:01.110321 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:01.652017 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:01.651980 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm" Oct 30 21:49:01.677888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:01.677860 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:01.738658 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:01.738297 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0c8943-d0ef-4c44-8757-e25e5c8b514e-kube-api-access-t52tr\" (UniqueName: \"kubernetes.io/projected/7e0c8943-d0ef-4c44-8757-e25e5c8b514e-kube-api-access-t52tr\") pod \"7e0c8943-d0ef-4c44-8757-e25e5c8b514e\" (UID: \"7e0c8943-d0ef-4c44-8757-e25e5c8b514e\") " Oct 30 21:49:01.739448 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:01.739417 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e0c8943-d0ef-4c44-8757-e25e5c8b514e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0c8943-d0ef-4c44-8757-e25e5c8b514e-maas-api-openshift-ca\") pod \"7e0c8943-d0ef-4c44-8757-e25e5c8b514e\" (UID: \"7e0c8943-d0ef-4c44-8757-e25e5c8b514e\") " Oct 30 21:49:01.739635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:01.739617 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e0c8943-d0ef-4c44-8757-e25e5c8b514e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0c8943-d0ef-4c44-8757-e25e5c8b514e-maas-api-xks-ca\") pod \"7e0c8943-d0ef-4c44-8757-e25e5c8b514e\" (UID: \"7e0c8943-d0ef-4c44-8757-e25e5c8b514e\") " Oct 30 21:49:01.740588 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:01.740532 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0c8943-d0ef-4c44-8757-e25e5c8b514e-maas-api-openshift-ca" pod "7e0c8943-d0ef-4c44-8757-e25e5c8b514e" (UID: "7e0c8943-d0ef-4c44-8757-e25e5c8b514e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:01.740964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:01.740944 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0c8943-d0ef-4c44-8757-e25e5c8b514e-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:01.748886 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:01.748856 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0c8943-d0ef-4c44-8757-e25e5c8b514e-maas-api-xks-ca" pod "7e0c8943-d0ef-4c44-8757-e25e5c8b514e" (UID: "7e0c8943-d0ef-4c44-8757-e25e5c8b514e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:01.760772 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:01.752978 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0c8943-d0ef-4c44-8757-e25e5c8b514e-kube-api-access-t52tr" pod "7e0c8943-d0ef-4c44-8757-e25e5c8b514e" (UID: "7e0c8943-d0ef-4c44-8757-e25e5c8b514e"). InnerVolumeSpecName "kube-api-access-t52tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:01.843131 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:01.843100 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t52tr\" (UniqueName: \"kubernetes.io/projected/7e0c8943-d0ef-4c44-8757-e25e5c8b514e-kube-api-access-t52tr\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:01.843131 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:01.843129 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0c8943-d0ef-4c44-8757-e25e5c8b514e-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:02.200470 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:02.200387 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm" Oct 30 21:49:02.200470 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:02.200432 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm" event={"ID":"7e0c8943-d0ef-4c44-8757-e25e5c8b514e","Type":"ContainerDied","Data":"29eaeb88c0124899d5e58dab83d935335ba9bdabe70210e2ed73acb5207449b7"} Oct 30 21:49:02.201326 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:02.200478 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29eaeb88c0124899d5e58dab83d935335ba9bdabe70210e2ed73acb5207449b7" Oct 30 21:49:02.347549 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:02.347509 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:03.044405 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:03.044365 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:03.308061 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:03.308019 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:03.547232 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:03.547195 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:03.981433 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:03.981401 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:04.660591 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:04.660558 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:05.125910 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:05.125854 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl" Oct 30 21:49:05.126094 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:05.125945 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j" Oct 30 21:49:05.451948 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:05.451864 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:05.936244 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:05.936173 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79" podUID="c7b2ea58-b940-4434-8b89-c3d7c0bc3cba" containerName="payload-processing" containerID="cri-o://12de89c06f58497c1090ae71ea18c1f56245e65d5e1ab8ab7266403227676ca4" gracePeriod=25 Oct 30 21:49:06.155746 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:06.155713 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:06.411914 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:06.411879 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8"] Oct 30 21:49:06.479331 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:06.479237 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:06.581102 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:06.581071 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:07.167057 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:07.167021 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:07.933507 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:07.933474 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:08.270545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:08.270461 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j"] Oct 30 21:49:08.282640 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:08.282600 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd"] Oct 30 21:49:08.336386 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:08.336346 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:08.436007 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:08.435966 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:08.663464 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:08.663425 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:09.532809 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:09.532773 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:09.723427 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:09.723388 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:09.828497 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:09.828462 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:10.199085 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:10.198997 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc"] Oct 30 21:49:10.199232 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:10.199211 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" podUID="07fab6bc-5ee9-4724-aa10-dcf0df1af13b" containerName="istio-proxy" containerID="cri-o://a19a452c99d9201e48336b7e0e807f5096f517fe29e1f9c6137146695c1bcaee" gracePeriod=30 Oct 30 21:49:10.205539 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:10.205504 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r"] Oct 30 21:49:10.205723 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:10.205702 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" podUID="db7b6603-3dfd-4f20-a362-da44b2fb9da9" containerName="istio-proxy" containerID="cri-o://d9e6af01d26650702e74d0f2291aa5472f8bc11ab3426247d2970ea56202f1cb" gracePeriod=30 Oct 30 21:49:10.211678 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:10.211646 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89"] Oct 30 21:49:10.211876 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:10.211853 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" podUID="630b8106-55a0-46e3-9de9-85898b01255d" containerName="istio-proxy" containerID="cri-o://17f889ac80e2685467e87c968ee179f97b0ab247d9498a53dce650ea89780c92" gracePeriod=30 Oct 30 21:49:10.491685 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:10.491562 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" containerID="cri-o://f8a0946cab502355d7673a363fc17963df2aac28f3a8806e2607ad696c6161c0" gracePeriod=45 Oct 30 21:49:10.523610 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:10.513935 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" containerID="cri-o://332ba3923ba97d6d3b413bf08d3f7d3e5f88bac7a64dccb08426f780e8052e6b" gracePeriod=45 Oct 30 21:49:10.524991 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:10.524823 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" containerID="cri-o://78761f0cd4b28ffd158c5476b75a143d7d8bfed44c4cfb3b7d175d1198ef3e82" gracePeriod=45 Oct 30 21:49:10.663340 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:10.663281 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:10.890341 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:10.890314 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:49:10.934502 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:10.934477 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:49:10.938660 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:10.938637 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:49:11.067432 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.067405 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-model-cache\") pod \"2f352e82-4be6-4c2c-bda0-bc109f65b739\" (UID: \"2f352e82-4be6-4c2c-bda0-bc109f65b739\") " Oct 30 21:49:11.067596 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.067507 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a810e8ce-0559-414c-a5ee-eafb7822dd6a-tls-certs\" (UniqueName: \"kubernetes.io/secret/a810e8ce-0559-414c-a5ee-eafb7822dd6a-tls-certs\") pod \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\" (UID: \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\") " Oct 30 21:49:11.067596 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.067530 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a810e8ce-0559-414c-a5ee-eafb7822dd6a-kube-api-access-kv2z4\" (UniqueName: \"kubernetes.io/projected/a810e8ce-0559-414c-a5ee-eafb7822dd6a-kube-api-access-kv2z4\") pod \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\" (UID: \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\") " Oct 30 21:49:11.067596 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.067564 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-model-cache\") pod \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\" (UID: \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\") " Oct 30 21:49:11.067769 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.067673 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-model-cache" pod "2f352e82-4be6-4c2c-bda0-bc109f65b739" (UID: "2f352e82-4be6-4c2c-bda0-bc109f65b739"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:11.067769 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.067714 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-tmp-dir\") pod \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\" (UID: \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\") " Oct 30 21:49:11.067879 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.067788 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-dshm\") pod \"2f352e82-4be6-4c2c-bda0-bc109f65b739\" (UID: \"2f352e82-4be6-4c2c-bda0-bc109f65b739\") " Oct 30 21:49:11.067879 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.067832 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-model-cache" pod "a810e8ce-0559-414c-a5ee-eafb7822dd6a" (UID: "a810e8ce-0559-414c-a5ee-eafb7822dd6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:11.067879 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.067841 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-kube-api-access-2cqbd\" (UniqueName: \"kubernetes.io/projected/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-kube-api-access-2cqbd\") pod \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\" (UID: \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\") " Oct 30 21:49:11.068027 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.067900 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-home\" (UniqueName: \"kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-home\") pod \"2f352e82-4be6-4c2c-bda0-bc109f65b739\" (UID: \"2f352e82-4be6-4c2c-bda0-bc109f65b739\") " Oct 30 21:49:11.068027 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.067965 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-tmp-dir" pod "986c5f2e-ccbb-4d9f-9773-e83e7de17fff" (UID: "986c5f2e-ccbb-4d9f-9773-e83e7de17fff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:11.068135 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.068033 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-dshm\") pod \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\" (UID: \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\") " Oct 30 21:49:11.068135 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.068065 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-tmp-dir\") pod \"2f352e82-4be6-4c2c-bda0-bc109f65b739\" (UID: \"2f352e82-4be6-4c2c-bda0-bc109f65b739\") " Oct 30 21:49:11.068240 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.068137 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-model-cache\") pod \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\" (UID: \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\") " Oct 30 21:49:11.068240 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.068180 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f352e82-4be6-4c2c-bda0-bc109f65b739-kube-api-access-bxx4b\" (UniqueName: \"kubernetes.io/projected/2f352e82-4be6-4c2c-bda0-bc109f65b739-kube-api-access-bxx4b\") pod \"2f352e82-4be6-4c2c-bda0-bc109f65b739\" (UID: \"2f352e82-4be6-4c2c-bda0-bc109f65b739\") " Oct 30 21:49:11.068240 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.068227 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-tmp-dir\") pod \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\" (UID: \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\") " Oct 30 21:49:11.068384 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.068316 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-home\" (UniqueName: \"kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-home\") pod \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\" (UID: \"a810e8ce-0559-414c-a5ee-eafb7822dd6a\") " Oct 30 21:49:11.068384 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.068359 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-tmp-dir" pod "2f352e82-4be6-4c2c-bda0-bc109f65b739" (UID: "2f352e82-4be6-4c2c-bda0-bc109f65b739"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:11.068384 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.068372 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-tls-certs\" (UniqueName: \"kubernetes.io/secret/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-tls-certs\") pod \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\" (UID: \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\") " Oct 30 21:49:11.068542 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.068419 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-home\" (UniqueName: \"kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-home\") pod \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\" (UID: \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\") " Oct 30 21:49:11.068542 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.068450 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-model-cache" pod "986c5f2e-ccbb-4d9f-9773-e83e7de17fff" (UID: "986c5f2e-ccbb-4d9f-9773-e83e7de17fff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:11.068542 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.068468 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-dshm\") pod \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\" (UID: \"986c5f2e-ccbb-4d9f-9773-e83e7de17fff\") " Oct 30 21:49:11.068542 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.068496 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f352e82-4be6-4c2c-bda0-bc109f65b739-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f352e82-4be6-4c2c-bda0-bc109f65b739-tls-certs\") pod \"2f352e82-4be6-4c2c-bda0-bc109f65b739\" (UID: \"2f352e82-4be6-4c2c-bda0-bc109f65b739\") " Oct 30 21:49:11.068728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.068610 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-tmp-dir" pod "a810e8ce-0559-414c-a5ee-eafb7822dd6a" (UID: "a810e8ce-0559-414c-a5ee-eafb7822dd6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:11.069507 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.068981 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-home" pod "a810e8ce-0559-414c-a5ee-eafb7822dd6a" (UID: "a810e8ce-0559-414c-a5ee-eafb7822dd6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:11.069507 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.069051 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-home" pod "986c5f2e-ccbb-4d9f-9773-e83e7de17fff" (UID: "986c5f2e-ccbb-4d9f-9773-e83e7de17fff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:11.069507 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.069372 5801 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:11.069507 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.069394 5801 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:11.069507 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.069410 5801 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:11.069507 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.069425 5801 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:11.069507 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.069440 5801 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:11.069507 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.069455 5801 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:11.069507 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.069468 5801 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:11.069507 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.069482 5801 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:11.070350 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.070321 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a810e8ce-0559-414c-a5ee-eafb7822dd6a-tls-certs" pod "a810e8ce-0559-414c-a5ee-eafb7822dd6a" (UID: "a810e8ce-0559-414c-a5ee-eafb7822dd6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:11.070410 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.070357 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-dshm" pod "a810e8ce-0559-414c-a5ee-eafb7822dd6a" (UID: "a810e8ce-0559-414c-a5ee-eafb7822dd6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:11.070565 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.070537 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-home" pod "2f352e82-4be6-4c2c-bda0-bc109f65b739" (UID: "2f352e82-4be6-4c2c-bda0-bc109f65b739"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:11.070928 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.070904 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-dshm" pod "2f352e82-4be6-4c2c-bda0-bc109f65b739" (UID: "2f352e82-4be6-4c2c-bda0-bc109f65b739"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:11.071046 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.070938 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-kube-api-access-2cqbd" pod "986c5f2e-ccbb-4d9f-9773-e83e7de17fff" (UID: "986c5f2e-ccbb-4d9f-9773-e83e7de17fff"). InnerVolumeSpecName "kube-api-access-2cqbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:11.071207 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.071182 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a810e8ce-0559-414c-a5ee-eafb7822dd6a-kube-api-access-kv2z4" pod "a810e8ce-0559-414c-a5ee-eafb7822dd6a" (UID: "a810e8ce-0559-414c-a5ee-eafb7822dd6a"). InnerVolumeSpecName "kube-api-access-kv2z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:11.071414 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.071394 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f352e82-4be6-4c2c-bda0-bc109f65b739-tls-certs" pod "2f352e82-4be6-4c2c-bda0-bc109f65b739" (UID: "2f352e82-4be6-4c2c-bda0-bc109f65b739"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:11.071488 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.071389 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-dshm" pod "986c5f2e-ccbb-4d9f-9773-e83e7de17fff" (UID: "986c5f2e-ccbb-4d9f-9773-e83e7de17fff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:11.072081 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.072061 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f352e82-4be6-4c2c-bda0-bc109f65b739-kube-api-access-bxx4b" pod "2f352e82-4be6-4c2c-bda0-bc109f65b739" (UID: "2f352e82-4be6-4c2c-bda0-bc109f65b739"). InnerVolumeSpecName "kube-api-access-bxx4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:11.072395 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.072374 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-tls-certs" pod "986c5f2e-ccbb-4d9f-9773-e83e7de17fff" (UID: "986c5f2e-ccbb-4d9f-9773-e83e7de17fff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:11.171285 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.171254 5801 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a810e8ce-0559-414c-a5ee-eafb7822dd6a-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:11.171285 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.171283 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv2z4\" (UniqueName: \"kubernetes.io/projected/a810e8ce-0559-414c-a5ee-eafb7822dd6a-kube-api-access-kv2z4\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:11.171285 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.171293 5801 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:11.171526 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.171303 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cqbd\" (UniqueName: \"kubernetes.io/projected/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-kube-api-access-2cqbd\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:11.171526 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.171311 5801 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f352e82-4be6-4c2c-bda0-bc109f65b739-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:11.171526 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.171319 5801 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a810e8ce-0559-414c-a5ee-eafb7822dd6a-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:11.171526 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.171328 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxx4b\" (UniqueName: \"kubernetes.io/projected/2f352e82-4be6-4c2c-bda0-bc109f65b739-kube-api-access-bxx4b\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:11.171526 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.171336 5801 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:11.171526 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.171343 5801 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/986c5f2e-ccbb-4d9f-9773-e83e7de17fff-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:11.171526 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.171352 5801 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f352e82-4be6-4c2c-bda0-bc109f65b739-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:11.297988 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.297954 5801 generic.go:356] "Generic (PLEG): container finished" podID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerID="332ba3923ba97d6d3b413bf08d3f7d3e5f88bac7a64dccb08426f780e8052e6b" exitCode=0 Oct 30 21:49:11.298154 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.298011 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" Oct 30 21:49:11.298154 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.298038 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" event={"ID":"2f352e82-4be6-4c2c-bda0-bc109f65b739","Type":"ContainerDied","Data":"332ba3923ba97d6d3b413bf08d3f7d3e5f88bac7a64dccb08426f780e8052e6b"} Oct 30 21:49:11.298154 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.298083 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n" event={"ID":"2f352e82-4be6-4c2c-bda0-bc109f65b739","Type":"ContainerDied","Data":"c9702f18e79b51f9eaad13016ec979b4538b3005c34ba55dbff24eec063fbaf4"} Oct 30 21:49:11.298154 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.298108 5801 scope.go:122] "RemoveContainer" containerID="332ba3923ba97d6d3b413bf08d3f7d3e5f88bac7a64dccb08426f780e8052e6b" Oct 30 21:49:11.299660 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.299627 5801 generic.go:356] "Generic (PLEG): container finished" podID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerID="78761f0cd4b28ffd158c5476b75a143d7d8bfed44c4cfb3b7d175d1198ef3e82" exitCode=0 Oct 30 21:49:11.299807 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.299666 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" Oct 30 21:49:11.299807 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.299686 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" event={"ID":"986c5f2e-ccbb-4d9f-9773-e83e7de17fff","Type":"ContainerDied","Data":"78761f0cd4b28ffd158c5476b75a143d7d8bfed44c4cfb3b7d175d1198ef3e82"} Oct 30 21:49:11.299807 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.299722 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2" event={"ID":"986c5f2e-ccbb-4d9f-9773-e83e7de17fff","Type":"ContainerDied","Data":"32ebeabc1d0391046f9079c4f9b5e2a5e334e0281aad200da2befa9481c93782"} Oct 30 21:49:11.301540 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.301516 5801 generic.go:356] "Generic (PLEG): container finished" podID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerID="f8a0946cab502355d7673a363fc17963df2aac28f3a8806e2607ad696c6161c0" exitCode=0 Oct 30 21:49:11.301638 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.301569 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" Oct 30 21:49:11.301638 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.301577 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" event={"ID":"a810e8ce-0559-414c-a5ee-eafb7822dd6a","Type":"ContainerDied","Data":"f8a0946cab502355d7673a363fc17963df2aac28f3a8806e2607ad696c6161c0"} Oct 30 21:49:11.301638 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.301603 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw" event={"ID":"a810e8ce-0559-414c-a5ee-eafb7822dd6a","Type":"ContainerDied","Data":"65d9a05f47909c22d6802edd99c9e65550c585348fbedfd42a39db3afd183729"} Oct 30 21:49:11.312287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.312266 5801 scope.go:122] "RemoveContainer" containerID="332ba3923ba97d6d3b413bf08d3f7d3e5f88bac7a64dccb08426f780e8052e6b" Oct 30 21:49:11.312568 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:49:11.312546 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"332ba3923ba97d6d3b413bf08d3f7d3e5f88bac7a64dccb08426f780e8052e6b\": container with ID starting with 332ba3923ba97d6d3b413bf08d3f7d3e5f88bac7a64dccb08426f780e8052e6b not found: ID does not exist" containerID="332ba3923ba97d6d3b413bf08d3f7d3e5f88bac7a64dccb08426f780e8052e6b" Oct 30 21:49:11.312674 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.312575 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"332ba3923ba97d6d3b413bf08d3f7d3e5f88bac7a64dccb08426f780e8052e6b"} err="failed to get container status \"332ba3923ba97d6d3b413bf08d3f7d3e5f88bac7a64dccb08426f780e8052e6b\": rpc error: code = NotFound desc = could not find container \"332ba3923ba97d6d3b413bf08d3f7d3e5f88bac7a64dccb08426f780e8052e6b\": container with ID starting with 332ba3923ba97d6d3b413bf08d3f7d3e5f88bac7a64dccb08426f780e8052e6b not found: ID does not exist" Oct 30 21:49:11.312674 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.312592 5801 scope.go:122] "RemoveContainer" containerID="78761f0cd4b28ffd158c5476b75a143d7d8bfed44c4cfb3b7d175d1198ef3e82" Oct 30 21:49:11.322655 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.322640 5801 scope.go:122] "RemoveContainer" containerID="78761f0cd4b28ffd158c5476b75a143d7d8bfed44c4cfb3b7d175d1198ef3e82" Oct 30 21:49:11.322967 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:49:11.322943 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78761f0cd4b28ffd158c5476b75a143d7d8bfed44c4cfb3b7d175d1198ef3e82\": container with ID starting with 78761f0cd4b28ffd158c5476b75a143d7d8bfed44c4cfb3b7d175d1198ef3e82 not found: ID does not exist" containerID="78761f0cd4b28ffd158c5476b75a143d7d8bfed44c4cfb3b7d175d1198ef3e82" Oct 30 21:49:11.323030 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.322973 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78761f0cd4b28ffd158c5476b75a143d7d8bfed44c4cfb3b7d175d1198ef3e82"} err="failed to get container status \"78761f0cd4b28ffd158c5476b75a143d7d8bfed44c4cfb3b7d175d1198ef3e82\": rpc error: code = NotFound desc = could not find container \"78761f0cd4b28ffd158c5476b75a143d7d8bfed44c4cfb3b7d175d1198ef3e82\": container with ID starting with 78761f0cd4b28ffd158c5476b75a143d7d8bfed44c4cfb3b7d175d1198ef3e82 not found: ID does not exist" Oct 30 21:49:11.323030 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.322988 5801 scope.go:122] "RemoveContainer" containerID="f8a0946cab502355d7673a363fc17963df2aac28f3a8806e2607ad696c6161c0" Oct 30 21:49:11.334721 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.334700 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2"] Oct 30 21:49:11.337011 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.336990 5801 scope.go:122] "RemoveContainer" containerID="f8a0946cab502355d7673a363fc17963df2aac28f3a8806e2607ad696c6161c0" Oct 30 21:49:11.337304 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:49:11.337281 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8a0946cab502355d7673a363fc17963df2aac28f3a8806e2607ad696c6161c0\": container with ID starting with f8a0946cab502355d7673a363fc17963df2aac28f3a8806e2607ad696c6161c0 not found: ID does not exist" containerID="f8a0946cab502355d7673a363fc17963df2aac28f3a8806e2607ad696c6161c0" Oct 30 21:49:11.337413 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.337312 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8a0946cab502355d7673a363fc17963df2aac28f3a8806e2607ad696c6161c0"} err="failed to get container status \"f8a0946cab502355d7673a363fc17963df2aac28f3a8806e2607ad696c6161c0\": rpc error: code = NotFound desc = could not find container \"f8a0946cab502355d7673a363fc17963df2aac28f3a8806e2607ad696c6161c0\": container with ID starting with f8a0946cab502355d7673a363fc17963df2aac28f3a8806e2607ad696c6161c0 not found: ID does not exist" Oct 30 21:49:11.340603 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.340583 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-7dfff0/simulator-w6-7dfff0-kserve-86b668d6b-6g2c2"] Oct 30 21:49:11.352006 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.351980 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n"] Oct 30 21:49:11.356402 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.356383 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-7dfff0/premium-w6-7dfff0-kserve-bc6bc9747-t8g7n"] Oct 30 21:49:11.366531 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.366499 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw"] Oct 30 21:49:11.371031 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.371012 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-7dfff0/distinct-2-w6-7dfff0-kserve-6f954c7857-654qw"] Oct 30 21:49:11.412271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.412247 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8" podUID="6d340f73-5c9e-49d7-8703-5043db3c6193" containerName="payload-pre-processing" containerID="cri-o://d27197f7ab67ad7eb68692965d4d5fd8eb7eaf070db6ba407fb3afa96d2b9979" gracePeriod=25 Oct 30 21:49:11.454388 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.454355 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:11.533723 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.533697 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" path="/var/lib/kubelet/pods/2f352e82-4be6-4c2c-bda0-bc109f65b739/volumes" Oct 30 21:49:11.534335 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.534317 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" path="/var/lib/kubelet/pods/986c5f2e-ccbb-4d9f-9773-e83e7de17fff/volumes" Oct 30 21:49:11.534841 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.534823 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" path="/var/lib/kubelet/pods/a810e8ce-0559-414c-a5ee-eafb7822dd6a/volumes" Oct 30 21:49:11.734709 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.734682 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8" Oct 30 21:49:11.781845 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.781816 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d340f73-5c9e-49d7-8703-5043db3c6193-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d340f73-5c9e-49d7-8703-5043db3c6193-plugins-config-volume\") pod \"6d340f73-5c9e-49d7-8703-5043db3c6193\" (UID: \"6d340f73-5c9e-49d7-8703-5043db3c6193\") " Oct 30 21:49:11.782003 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.781963 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d340f73-5c9e-49d7-8703-5043db3c6193-kube-api-access-jbtk5\" (UniqueName: \"kubernetes.io/projected/6d340f73-5c9e-49d7-8703-5043db3c6193-kube-api-access-jbtk5\") pod \"6d340f73-5c9e-49d7-8703-5043db3c6193\" (UID: \"6d340f73-5c9e-49d7-8703-5043db3c6193\") " Oct 30 21:49:11.782300 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.782267 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d340f73-5c9e-49d7-8703-5043db3c6193-plugins-config-volume" pod "6d340f73-5c9e-49d7-8703-5043db3c6193" (UID: "6d340f73-5c9e-49d7-8703-5043db3c6193"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:11.782892 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.782869 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d340f73-5c9e-49d7-8703-5043db3c6193-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:11.784095 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.784074 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d340f73-5c9e-49d7-8703-5043db3c6193-kube-api-access-jbtk5" pod "6d340f73-5c9e-49d7-8703-5043db3c6193" (UID: "6d340f73-5c9e-49d7-8703-5043db3c6193"). InnerVolumeSpecName "kube-api-access-jbtk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:11.884350 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:11.884271 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbtk5\" (UniqueName: \"kubernetes.io/projected/6d340f73-5c9e-49d7-8703-5043db3c6193-kube-api-access-jbtk5\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:12.269321 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.269230 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:12.321037 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.319873 5801 generic.go:356] "Generic (PLEG): container finished" podID="6d340f73-5c9e-49d7-8703-5043db3c6193" containerID="d27197f7ab67ad7eb68692965d4d5fd8eb7eaf070db6ba407fb3afa96d2b9979" exitCode=0 Oct 30 21:49:12.321037 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.320437 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8" Oct 30 21:49:12.321037 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.320506 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8" event={"ID":"6d340f73-5c9e-49d7-8703-5043db3c6193","Type":"ContainerDied","Data":"d27197f7ab67ad7eb68692965d4d5fd8eb7eaf070db6ba407fb3afa96d2b9979"} Oct 30 21:49:12.321037 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.320598 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8" event={"ID":"6d340f73-5c9e-49d7-8703-5043db3c6193","Type":"ContainerDied","Data":"469fa407371979c515993134becedf22d4a9891c3acc948a110620ad61a528f7"} Oct 30 21:49:12.321037 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.320628 5801 scope.go:122] "RemoveContainer" containerID="d27197f7ab67ad7eb68692965d4d5fd8eb7eaf070db6ba407fb3afa96d2b9979" Oct 30 21:49:12.335109 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.334842 5801 scope.go:122] "RemoveContainer" containerID="d27197f7ab67ad7eb68692965d4d5fd8eb7eaf070db6ba407fb3afa96d2b9979" Oct 30 21:49:12.335201 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:49:12.335158 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d27197f7ab67ad7eb68692965d4d5fd8eb7eaf070db6ba407fb3afa96d2b9979\": container with ID starting with d27197f7ab67ad7eb68692965d4d5fd8eb7eaf070db6ba407fb3afa96d2b9979 not found: ID does not exist" containerID="d27197f7ab67ad7eb68692965d4d5fd8eb7eaf070db6ba407fb3afa96d2b9979" Oct 30 21:49:12.335268 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.335193 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d27197f7ab67ad7eb68692965d4d5fd8eb7eaf070db6ba407fb3afa96d2b9979"} err="failed to get container status \"d27197f7ab67ad7eb68692965d4d5fd8eb7eaf070db6ba407fb3afa96d2b9979\": rpc error: code = NotFound desc = could not find container \"d27197f7ab67ad7eb68692965d4d5fd8eb7eaf070db6ba407fb3afa96d2b9979\": container with ID starting with d27197f7ab67ad7eb68692965d4d5fd8eb7eaf070db6ba407fb3afa96d2b9979 not found: ID does not exist" Oct 30 21:49:12.349776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.349734 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8"] Oct 30 21:49:12.355357 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.355332 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08aafeac-65b768659-77rx8"] Oct 30 21:49:12.759174 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.759105 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:12.782824 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.782792 5801 patch_prober.go:29] interesting pod/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:49:12.783018 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.782857 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" podUID="07fab6bc-5ee9-4724-aa10-dcf0df1af13b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:49:12.923209 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.923131 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb"] Oct 30 21:49:12.923893 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.923876 5801 cpu_manager.go:414] "containerMap: removing container" podUID="7e0c8943-d0ef-4c44-8757-e25e5c8b514e" containerName="revoke-keys" Oct 30 21:49:12.923959 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.923894 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0c8943-d0ef-4c44-8757-e25e5c8b514e" containerName="revoke-keys" Oct 30 21:49:12.923959 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.923907 5801 cpu_manager.go:414] "containerMap: removing container" podUID="6d340f73-5c9e-49d7-8703-5043db3c6193" containerName="payload-pre-processing" Oct 30 21:49:12.923959 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.923912 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d340f73-5c9e-49d7-8703-5043db3c6193" containerName="payload-pre-processing" Oct 30 21:49:12.923959 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.923945 5801 cpu_manager.go:414] "containerMap: removing container" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" Oct 30 21:49:12.923959 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.923950 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" Oct 30 21:49:12.924111 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.923962 5801 cpu_manager.go:414] "containerMap: removing container" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" Oct 30 21:49:12.924111 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.923967 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" Oct 30 21:49:12.924111 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.923976 5801 cpu_manager.go:414] "containerMap: removing container" podUID="b3ef08c1-b100-409a-ab98-f36965c0c826" containerName="revoke-keys" Oct 30 21:49:12.924111 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.923981 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3ef08c1-b100-409a-ab98-f36965c0c826" containerName="revoke-keys" Oct 30 21:49:12.924111 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.924007 5801 cpu_manager.go:414] "containerMap: removing container" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" Oct 30 21:49:12.924111 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.924012 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" Oct 30 21:49:12.924111 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.924106 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f352e82-4be6-4c2c-bda0-bc109f65b739" containerName="main" Oct 30 21:49:12.924314 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.924115 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0c8943-d0ef-4c44-8757-e25e5c8b514e" containerName="revoke-keys" Oct 30 21:49:12.924314 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.924122 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d340f73-5c9e-49d7-8703-5043db3c6193" containerName="payload-pre-processing" Oct 30 21:49:12.924314 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.924134 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="986c5f2e-ccbb-4d9f-9773-e83e7de17fff" containerName="main" Oct 30 21:49:12.924314 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.924141 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="a810e8ce-0559-414c-a5ee-eafb7822dd6a" containerName="main" Oct 30 21:49:12.924314 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.924147 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3ef08c1-b100-409a-ab98-f36965c0c826" containerName="revoke-keys" Oct 30 21:49:12.929221 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.929199 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb" Oct 30 21:49:12.933138 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.933113 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:49:12.933279 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.933224 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:49:12.933279 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.933256 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:49:12.934103 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.934084 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:49:12.936332 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.936276 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:12.937776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:12.937741 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb"] Oct 30 21:49:13.009505 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.009460 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb\" (UID: \"9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb" Oct 30 21:49:13.009713 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.009541 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwpvq\" (UniqueName: \"kubernetes.io/projected/9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9-kube-api-access-qwpvq\") pod \"maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb\" (UID: \"9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb" Oct 30 21:49:13.010043 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.010013 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb\" (UID: \"9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb" Oct 30 21:49:13.111668 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.111635 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb\" (UID: \"9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb" Oct 30 21:49:13.111832 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.111777 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb\" (UID: \"9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb" Oct 30 21:49:13.111904 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.111837 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwpvq\" (UniqueName: \"kubernetes.io/projected/9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9-kube-api-access-qwpvq\") pod \"maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb\" (UID: \"9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb" Oct 30 21:49:13.112459 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.112433 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb\" (UID: \"9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb" Oct 30 21:49:13.114332 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.114307 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb\" (UID: \"9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb" Oct 30 21:49:13.125166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.125137 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwpvq\" (UniqueName: \"kubernetes.io/projected/9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9-kube-api-access-qwpvq\") pod \"maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb\" (UID: \"9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb" Oct 30 21:49:13.247481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.247398 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb" Oct 30 21:49:13.271743 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.271707 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j" podUID="8096ecd8-04a2-493d-b0e4-66997cdd6d6f" containerName="payload-processing" containerID="cri-o://5de17218b2baf1488d1fd8c529e2fdb0efc01fdca1f6329f99583b238f3aee5e" gracePeriod=25 Oct 30 21:49:13.283388 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.283347 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd" podUID="de103d6d-3060-401a-9d10-990b38101c62" containerName="payload-pre-processing" containerID="cri-o://caf53e8a9d60618c035abe46943ad87e4775a7f6389997ae64837e3540ce6aba" gracePeriod=25 Oct 30 21:49:13.538631 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.538601 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d340f73-5c9e-49d7-8703-5043db3c6193" path="/var/lib/kubelet/pods/6d340f73-5c9e-49d7-8703-5043db3c6193/volumes" Oct 30 21:49:13.540259 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:49:13.540229 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb45b98_bff7_4fc8_93b3_bb68fb9c2ed9.slice/crio-08f2371370490d348866e556ef1142e111e13f184b512967cf6a63f49acda952 WatchSource:0}: Error finding container 08f2371370490d348866e556ef1142e111e13f184b512967cf6a63f49acda952: Status 404 returned error can't find the container with id 08f2371370490d348866e556ef1142e111e13f184b512967cf6a63f49acda952 Oct 30 21:49:13.540403 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.540306 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb"] Oct 30 21:49:13.561107 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.561080 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j" Oct 30 21:49:13.614273 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.614246 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd" Oct 30 21:49:13.621584 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.621557 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8096ecd8-04a2-493d-b0e4-66997cdd6d6f-kube-api-access-gbn2q\" (UniqueName: \"kubernetes.io/projected/8096ecd8-04a2-493d-b0e4-66997cdd6d6f-kube-api-access-gbn2q\") pod \"8096ecd8-04a2-493d-b0e4-66997cdd6d6f\" (UID: \"8096ecd8-04a2-493d-b0e4-66997cdd6d6f\") " Oct 30 21:49:13.621720 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.621702 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8096ecd8-04a2-493d-b0e4-66997cdd6d6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8096ecd8-04a2-493d-b0e4-66997cdd6d6f-plugins-config-volume\") pod \"8096ecd8-04a2-493d-b0e4-66997cdd6d6f\" (UID: \"8096ecd8-04a2-493d-b0e4-66997cdd6d6f\") " Oct 30 21:49:13.622105 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.622080 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8096ecd8-04a2-493d-b0e4-66997cdd6d6f-plugins-config-volume" pod "8096ecd8-04a2-493d-b0e4-66997cdd6d6f" (UID: "8096ecd8-04a2-493d-b0e4-66997cdd6d6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:13.622370 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.622352 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8096ecd8-04a2-493d-b0e4-66997cdd6d6f-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:13.623766 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.623730 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8096ecd8-04a2-493d-b0e4-66997cdd6d6f-kube-api-access-gbn2q" pod "8096ecd8-04a2-493d-b0e4-66997cdd6d6f" (UID: "8096ecd8-04a2-493d-b0e4-66997cdd6d6f"). InnerVolumeSpecName "kube-api-access-gbn2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:13.723586 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.723554 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de103d6d-3060-401a-9d10-990b38101c62-kube-api-access-k99hp\" (UniqueName: \"kubernetes.io/projected/de103d6d-3060-401a-9d10-990b38101c62-kube-api-access-k99hp\") pod \"de103d6d-3060-401a-9d10-990b38101c62\" (UID: \"de103d6d-3060-401a-9d10-990b38101c62\") " Oct 30 21:49:13.723788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.723743 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de103d6d-3060-401a-9d10-990b38101c62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de103d6d-3060-401a-9d10-990b38101c62-plugins-config-volume\") pod \"de103d6d-3060-401a-9d10-990b38101c62\" (UID: \"de103d6d-3060-401a-9d10-990b38101c62\") " Oct 30 21:49:13.724118 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.724093 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de103d6d-3060-401a-9d10-990b38101c62-plugins-config-volume" pod "de103d6d-3060-401a-9d10-990b38101c62" (UID: "de103d6d-3060-401a-9d10-990b38101c62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:13.724358 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.724343 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbn2q\" (UniqueName: \"kubernetes.io/projected/8096ecd8-04a2-493d-b0e4-66997cdd6d6f-kube-api-access-gbn2q\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:13.724392 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.724362 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de103d6d-3060-401a-9d10-990b38101c62-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:13.725694 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.725676 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de103d6d-3060-401a-9d10-990b38101c62-kube-api-access-k99hp" pod "de103d6d-3060-401a-9d10-990b38101c62" (UID: "de103d6d-3060-401a-9d10-990b38101c62"). InnerVolumeSpecName "kube-api-access-k99hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:13.825902 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.825861 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k99hp\" (UniqueName: \"kubernetes.io/projected/de103d6d-3060-401a-9d10-990b38101c62-kube-api-access-k99hp\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:13.971958 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:13.971919 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79" podUID="c7b2ea58-b940-4434-8b89-c3d7c0bc3cba" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:49:14.131410 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.131374 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:14.345508 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.345471 5801 generic.go:356] "Generic (PLEG): container finished" podID="8096ecd8-04a2-493d-b0e4-66997cdd6d6f" containerID="5de17218b2baf1488d1fd8c529e2fdb0efc01fdca1f6329f99583b238f3aee5e" exitCode=0 Oct 30 21:49:14.345508 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.345510 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j" Oct 30 21:49:14.345727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.345552 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j" event={"ID":"8096ecd8-04a2-493d-b0e4-66997cdd6d6f","Type":"ContainerDied","Data":"5de17218b2baf1488d1fd8c529e2fdb0efc01fdca1f6329f99583b238f3aee5e"} Oct 30 21:49:14.345727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.345601 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j" event={"ID":"8096ecd8-04a2-493d-b0e4-66997cdd6d6f","Type":"ContainerDied","Data":"46a8fbccc526021a9ac1995ffeb084d10da33fb9ef46b5d293582ab584b45344"} Oct 30 21:49:14.345727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.345626 5801 scope.go:122] "RemoveContainer" containerID="5de17218b2baf1488d1fd8c529e2fdb0efc01fdca1f6329f99583b238f3aee5e" Oct 30 21:49:14.347212 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.347189 5801 generic.go:356] "Generic (PLEG): container finished" podID="de103d6d-3060-401a-9d10-990b38101c62" containerID="caf53e8a9d60618c035abe46943ad87e4775a7f6389997ae64837e3540ce6aba" exitCode=0 Oct 30 21:49:14.347317 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.347223 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd" Oct 30 21:49:14.347317 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.347271 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd" event={"ID":"de103d6d-3060-401a-9d10-990b38101c62","Type":"ContainerDied","Data":"caf53e8a9d60618c035abe46943ad87e4775a7f6389997ae64837e3540ce6aba"} Oct 30 21:49:14.347415 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.347332 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd" event={"ID":"de103d6d-3060-401a-9d10-990b38101c62","Type":"ContainerDied","Data":"9213a4598135309855dad01221ad733bad7544a5d4a82fee92677dbfa714227b"} Oct 30 21:49:14.349956 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.349436 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb" event={"ID":"9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9","Type":"ContainerStarted","Data":"e12414833d773583cf64631985773fa63158e3c882d2043ef6ed8f584ba4ef04"} Oct 30 21:49:14.349956 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.349480 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb" event={"ID":"9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9","Type":"ContainerStarted","Data":"08f2371370490d348866e556ef1142e111e13f184b512967cf6a63f49acda952"} Oct 30 21:49:14.358372 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.358351 5801 scope.go:122] "RemoveContainer" containerID="5de17218b2baf1488d1fd8c529e2fdb0efc01fdca1f6329f99583b238f3aee5e" Oct 30 21:49:14.358691 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:49:14.358670 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de17218b2baf1488d1fd8c529e2fdb0efc01fdca1f6329f99583b238f3aee5e\": container with ID starting with 5de17218b2baf1488d1fd8c529e2fdb0efc01fdca1f6329f99583b238f3aee5e not found: ID does not exist" containerID="5de17218b2baf1488d1fd8c529e2fdb0efc01fdca1f6329f99583b238f3aee5e" Oct 30 21:49:14.358807 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.358696 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de17218b2baf1488d1fd8c529e2fdb0efc01fdca1f6329f99583b238f3aee5e"} err="failed to get container status \"5de17218b2baf1488d1fd8c529e2fdb0efc01fdca1f6329f99583b238f3aee5e\": rpc error: code = NotFound desc = could not find container \"5de17218b2baf1488d1fd8c529e2fdb0efc01fdca1f6329f99583b238f3aee5e\": container with ID starting with 5de17218b2baf1488d1fd8c529e2fdb0efc01fdca1f6329f99583b238f3aee5e not found: ID does not exist" Oct 30 21:49:14.358807 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.358713 5801 scope.go:122] "RemoveContainer" containerID="caf53e8a9d60618c035abe46943ad87e4775a7f6389997ae64837e3540ce6aba" Oct 30 21:49:14.369102 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.369086 5801 scope.go:122] "RemoveContainer" containerID="caf53e8a9d60618c035abe46943ad87e4775a7f6389997ae64837e3540ce6aba" Oct 30 21:49:14.369357 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:49:14.369338 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caf53e8a9d60618c035abe46943ad87e4775a7f6389997ae64837e3540ce6aba\": container with ID starting with caf53e8a9d60618c035abe46943ad87e4775a7f6389997ae64837e3540ce6aba not found: ID does not exist" containerID="caf53e8a9d60618c035abe46943ad87e4775a7f6389997ae64837e3540ce6aba" Oct 30 21:49:14.369415 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.369363 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf53e8a9d60618c035abe46943ad87e4775a7f6389997ae64837e3540ce6aba"} err="failed to get container status \"caf53e8a9d60618c035abe46943ad87e4775a7f6389997ae64837e3540ce6aba\": rpc error: code = NotFound desc = could not find container \"caf53e8a9d60618c035abe46943ad87e4775a7f6389997ae64837e3540ce6aba\": container with ID starting with caf53e8a9d60618c035abe46943ad87e4775a7f6389997ae64837e3540ce6aba not found: ID does not exist" Oct 30 21:49:14.375847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.375809 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb" podStartSLOduration=2.375798059 podStartE2EDuration="2.375798059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:14.373409892 +0000 UTC m=+1222.969226554" watchObservedRunningTime="2026-10-30 21:49:14.375798059 +0000 UTC m=+1222.971614719" Oct 30 21:49:14.391937 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.391879 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j"] Oct 30 21:49:14.396472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.396449 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd5bf8-84dd447849-ml22j"] Oct 30 21:49:14.405177 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.405149 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd"] Oct 30 21:49:14.409419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.409399 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dd5bf8-5fbdd575-6qmbd"] Oct 30 21:49:14.889929 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.889895 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96ffa292-8b8gj"] Oct 30 21:49:14.890681 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.890665 5801 cpu_manager.go:414] "containerMap: removing container" podUID="de103d6d-3060-401a-9d10-990b38101c62" containerName="payload-pre-processing" Oct 30 21:49:14.890744 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.890682 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de103d6d-3060-401a-9d10-990b38101c62" containerName="payload-pre-processing" Oct 30 21:49:14.890744 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.890693 5801 cpu_manager.go:414] "containerMap: removing container" podUID="8096ecd8-04a2-493d-b0e4-66997cdd6d6f" containerName="payload-processing" Oct 30 21:49:14.890744 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.890698 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8096ecd8-04a2-493d-b0e4-66997cdd6d6f" containerName="payload-processing" Oct 30 21:49:14.890861 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.890833 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="de103d6d-3060-401a-9d10-990b38101c62" containerName="payload-pre-processing" Oct 30 21:49:14.890861 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.890844 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="8096ecd8-04a2-493d-b0e4-66997cdd6d6f" containerName="payload-processing" Oct 30 21:49:14.894609 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.894587 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ffa292-8b8gj" Oct 30 21:49:14.900114 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.900094 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63561e84-hqgzs"] Oct 30 21:49:14.909176 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.909150 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63561e84-hqgzs" Oct 30 21:49:14.924166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.924141 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96ffa292-8b8gj"] Oct 30 21:49:14.932704 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.932680 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63561e84-hqgzs"] Oct 30 21:49:14.943845 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.943815 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rjz2\" (UniqueName: \"kubernetes.io/projected/0d648b4b-c946-4b26-b77b-a3837e5b2135-kube-api-access-5rjz2\") pod \"maas-api-revoke-sub-63561e84-hqgzs\" (UID: \"0d648b4b-c946-4b26-b77b-a3837e5b2135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63561e84-hqgzs" Oct 30 21:49:14.944005 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.943982 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d648b4b-c946-4b26-b77b-a3837e5b2135-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63561e84-hqgzs\" (UID: \"0d648b4b-c946-4b26-b77b-a3837e5b2135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63561e84-hqgzs" Oct 30 21:49:14.944102 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.944084 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d648b4b-c946-4b26-b77b-a3837e5b2135-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63561e84-hqgzs\" (UID: \"0d648b4b-c946-4b26-b77b-a3837e5b2135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63561e84-hqgzs" Oct 30 21:49:14.944154 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.944141 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eecf577b-fd05-4675-b050-5cf510babaa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96ffa292-8b8gj\" (UID: \"eecf577b-fd05-4675-b050-5cf510babaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ffa292-8b8gj" Oct 30 21:49:14.944339 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.944316 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xts9g\" (UniqueName: \"kubernetes.io/projected/eecf577b-fd05-4675-b050-5cf510babaa4-kube-api-access-xts9g\") pod \"maas-api-revoke-sub-96ffa292-8b8gj\" (UID: \"eecf577b-fd05-4675-b050-5cf510babaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ffa292-8b8gj" Oct 30 21:49:14.944450 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:14.944434 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eecf577b-fd05-4675-b050-5cf510babaa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96ffa292-8b8gj\" (UID: \"eecf577b-fd05-4675-b050-5cf510babaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ffa292-8b8gj" Oct 30 21:49:15.045521 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.045484 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eecf577b-fd05-4675-b050-5cf510babaa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96ffa292-8b8gj\" (UID: \"eecf577b-fd05-4675-b050-5cf510babaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ffa292-8b8gj" Oct 30 21:49:15.045724 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.045622 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xts9g\" (UniqueName: \"kubernetes.io/projected/eecf577b-fd05-4675-b050-5cf510babaa4-kube-api-access-xts9g\") pod \"maas-api-revoke-sub-96ffa292-8b8gj\" (UID: \"eecf577b-fd05-4675-b050-5cf510babaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ffa292-8b8gj" Oct 30 21:49:15.045724 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.045713 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eecf577b-fd05-4675-b050-5cf510babaa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96ffa292-8b8gj\" (UID: \"eecf577b-fd05-4675-b050-5cf510babaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ffa292-8b8gj" Oct 30 21:49:15.045907 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.045889 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rjz2\" (UniqueName: \"kubernetes.io/projected/0d648b4b-c946-4b26-b77b-a3837e5b2135-kube-api-access-5rjz2\") pod \"maas-api-revoke-sub-63561e84-hqgzs\" (UID: \"0d648b4b-c946-4b26-b77b-a3837e5b2135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63561e84-hqgzs" Oct 30 21:49:15.046004 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.045989 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d648b4b-c946-4b26-b77b-a3837e5b2135-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63561e84-hqgzs\" (UID: \"0d648b4b-c946-4b26-b77b-a3837e5b2135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63561e84-hqgzs" Oct 30 21:49:15.046083 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.046052 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d648b4b-c946-4b26-b77b-a3837e5b2135-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63561e84-hqgzs\" (UID: \"0d648b4b-c946-4b26-b77b-a3837e5b2135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63561e84-hqgzs" Oct 30 21:49:15.046458 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.046433 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eecf577b-fd05-4675-b050-5cf510babaa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96ffa292-8b8gj\" (UID: \"eecf577b-fd05-4675-b050-5cf510babaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ffa292-8b8gj" Oct 30 21:49:15.046674 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.046656 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d648b4b-c946-4b26-b77b-a3837e5b2135-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63561e84-hqgzs\" (UID: \"0d648b4b-c946-4b26-b77b-a3837e5b2135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63561e84-hqgzs" Oct 30 21:49:15.048062 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.048037 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eecf577b-fd05-4675-b050-5cf510babaa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96ffa292-8b8gj\" (UID: \"eecf577b-fd05-4675-b050-5cf510babaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ffa292-8b8gj" Oct 30 21:49:15.048310 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.048292 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d648b4b-c946-4b26-b77b-a3837e5b2135-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63561e84-hqgzs\" (UID: \"0d648b4b-c946-4b26-b77b-a3837e5b2135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63561e84-hqgzs" Oct 30 21:49:15.057793 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.057770 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rjz2\" (UniqueName: \"kubernetes.io/projected/0d648b4b-c946-4b26-b77b-a3837e5b2135-kube-api-access-5rjz2\") pod \"maas-api-revoke-sub-63561e84-hqgzs\" (UID: \"0d648b4b-c946-4b26-b77b-a3837e5b2135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63561e84-hqgzs" Oct 30 21:49:15.058092 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.058071 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xts9g\" (UniqueName: \"kubernetes.io/projected/eecf577b-fd05-4675-b050-5cf510babaa4-kube-api-access-xts9g\") pod \"maas-api-revoke-sub-96ffa292-8b8gj\" (UID: \"eecf577b-fd05-4675-b050-5cf510babaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ffa292-8b8gj" Oct 30 21:49:15.212635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.212552 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ffa292-8b8gj" Oct 30 21:49:15.229687 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.229635 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63561e84-hqgzs" Oct 30 21:49:15.290293 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.290254 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:15.412692 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.399221 5801 generic.go:356] "Generic (PLEG): container finished" podID="630b8106-55a0-46e3-9de9-85898b01255d" containerID="17f889ac80e2685467e87c968ee179f97b0ab247d9498a53dce650ea89780c92" exitCode=0 Oct 30 21:49:15.412692 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.399320 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" event={"ID":"630b8106-55a0-46e3-9de9-85898b01255d","Type":"ContainerDied","Data":"17f889ac80e2685467e87c968ee179f97b0ab247d9498a53dce650ea89780c92"} Oct 30 21:49:15.444201 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.428795 5801 generic.go:356] "Generic (PLEG): container finished" podID="9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9" containerID="e12414833d773583cf64631985773fa63158e3c882d2043ef6ed8f584ba4ef04" exitCode=0 Oct 30 21:49:15.444201 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.429009 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb" event={"ID":"9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9","Type":"ContainerDied","Data":"e12414833d773583cf64631985773fa63158e3c882d2043ef6ed8f584ba4ef04"} Oct 30 21:49:15.444201 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.433737 5801 generic.go:356] "Generic (PLEG): container finished" podID="07fab6bc-5ee9-4724-aa10-dcf0df1af13b" containerID="a19a452c99d9201e48336b7e0e807f5096f517fe29e1f9c6137146695c1bcaee" exitCode=0 Oct 30 21:49:15.444201 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.433856 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" event={"ID":"07fab6bc-5ee9-4724-aa10-dcf0df1af13b","Type":"ContainerDied","Data":"a19a452c99d9201e48336b7e0e807f5096f517fe29e1f9c6137146695c1bcaee"} Oct 30 21:49:15.483235 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.482962 5801 generic.go:356] "Generic (PLEG): container finished" podID="db7b6603-3dfd-4f20-a362-da44b2fb9da9" containerID="d9e6af01d26650702e74d0f2291aa5472f8bc11ab3426247d2970ea56202f1cb" exitCode=0 Oct 30 21:49:15.483235 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.483087 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" event={"ID":"db7b6603-3dfd-4f20-a362-da44b2fb9da9","Type":"ContainerDied","Data":"d9e6af01d26650702e74d0f2291aa5472f8bc11ab3426247d2970ea56202f1cb"} Oct 30 21:49:15.532029 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.531999 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8096ecd8-04a2-493d-b0e4-66997cdd6d6f" path="/var/lib/kubelet/pods/8096ecd8-04a2-493d-b0e4-66997cdd6d6f/volumes" Oct 30 21:49:15.532823 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.532803 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de103d6d-3060-401a-9d10-990b38101c62" path="/var/lib/kubelet/pods/de103d6d-3060-401a-9d10-990b38101c62/volumes" Oct 30 21:49:15.659315 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.659283 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:49:15.712136 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.712007 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:49:15.717092 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.717068 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:49:15.718879 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.718854 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96ffa292-8b8gj"] Oct 30 21:49:15.719161 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:49:15.719143 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeecf577b_fd05_4675_b050_5cf510babaa4.slice/crio-ff6465c36003428bae3ae69cacab08f72df2c177d929759f8b07ffa94097793c WatchSource:0}: Error finding container ff6465c36003428bae3ae69cacab08f72df2c177d929759f8b07ffa94097793c: Status 404 returned error can't find the container with id ff6465c36003428bae3ae69cacab08f72df2c177d929759f8b07ffa94097793c Oct 30 21:49:15.741119 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.741090 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63561e84-hqgzs"] Oct 30 21:49:15.795211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.793977 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-workload-socket\") pod \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " Oct 30 21:49:15.795211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.794039 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istiod-ca-cert\") pod \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " Oct 30 21:49:15.795211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.794108 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-envoy\") pod \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " Oct 30 21:49:15.795211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.794173 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-token\" (UniqueName: \"kubernetes.io/projected/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-token\") pod \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " Oct 30 21:49:15.795211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.794222 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-workload-certs\") pod \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " Oct 30 21:49:15.795211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.794264 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-workload-socket\") pod \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " Oct 30 21:49:15.795211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.794316 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db7b6603-3dfd-4f20-a362-da44b2fb9da9-kube-api-access-nxq25\" (UniqueName: \"kubernetes.io/projected/db7b6603-3dfd-4f20-a362-da44b2fb9da9-kube-api-access-nxq25\") pod \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " Oct 30 21:49:15.795211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.794356 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-credential-socket\") pod \"630b8106-55a0-46e3-9de9-85898b01255d\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " Oct 30 21:49:15.795211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.794392 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/630b8106-55a0-46e3-9de9-85898b01255d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630b8106-55a0-46e3-9de9-85898b01255d-istiod-ca-cert\") pod \"630b8106-55a0-46e3-9de9-85898b01255d\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " Oct 30 21:49:15.795211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.794487 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-istio-envoy\") pod \"630b8106-55a0-46e3-9de9-85898b01255d\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " Oct 30 21:49:15.795211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.794567 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-workload-certs\") pod \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " Oct 30 21:49:15.795211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.794607 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-credential-socket\") pod \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " Oct 30 21:49:15.795211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.794670 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-podinfo\") pod \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " Oct 30 21:49:15.795211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.794708 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-workload-certs" pod "db7b6603-3dfd-4f20-a362-da44b2fb9da9" (UID: "db7b6603-3dfd-4f20-a362-da44b2fb9da9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:15.795211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.794720 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-token\" (UniqueName: \"kubernetes.io/projected/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-token\") pod \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " Oct 30 21:49:15.795211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.794842 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-workload-socket\") pod \"630b8106-55a0-46e3-9de9-85898b01255d\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " Oct 30 21:49:15.795211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.794879 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istiod-ca-cert\") pod \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " Oct 30 21:49:15.795211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.794918 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-istio-data\") pod \"630b8106-55a0-46e3-9de9-85898b01255d\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " Oct 30 21:49:15.795211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.794957 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-credential-socket\") pod \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " Oct 30 21:49:15.795211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.794978 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/630b8106-55a0-46e3-9de9-85898b01255d-kube-api-access-r9xx2\" (UniqueName: \"kubernetes.io/projected/630b8106-55a0-46e3-9de9-85898b01255d-kube-api-access-r9xx2\") pod \"630b8106-55a0-46e3-9de9-85898b01255d\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " Oct 30 21:49:15.795211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.795002 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-workload-certs\") pod \"630b8106-55a0-46e3-9de9-85898b01255d\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " Oct 30 21:49:15.795211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.795101 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-data\") pod \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " Oct 30 21:49:15.795211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.795150 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-kube-api-access-b8rpn\" (UniqueName: \"kubernetes.io/projected/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-kube-api-access-b8rpn\") pod \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\" (UID: \"07fab6bc-5ee9-4724-aa10-dcf0df1af13b\") " Oct 30 21:49:15.795211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.795218 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-data\") pod \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " Oct 30 21:49:15.797323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.795245 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-envoy\") pod \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " Oct 30 21:49:15.797323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.795304 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/630b8106-55a0-46e3-9de9-85898b01255d-istio-token\" (UniqueName: \"kubernetes.io/projected/630b8106-55a0-46e3-9de9-85898b01255d-istio-token\") pod \"630b8106-55a0-46e3-9de9-85898b01255d\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " Oct 30 21:49:15.797323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.795339 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/630b8106-55a0-46e3-9de9-85898b01255d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630b8106-55a0-46e3-9de9-85898b01255d-istio-podinfo\") pod \"630b8106-55a0-46e3-9de9-85898b01255d\" (UID: \"630b8106-55a0-46e3-9de9-85898b01255d\") " Oct 30 21:49:15.797323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.795374 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-podinfo\") pod \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\" (UID: \"db7b6603-3dfd-4f20-a362-da44b2fb9da9\") " Oct 30 21:49:15.797323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.796611 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.797323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.797103 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-workload-socket" pod "07fab6bc-5ee9-4724-aa10-dcf0df1af13b" (UID: "07fab6bc-5ee9-4724-aa10-dcf0df1af13b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:15.797642 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.797356 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-istio-data" pod "630b8106-55a0-46e3-9de9-85898b01255d" (UID: "630b8106-55a0-46e3-9de9-85898b01255d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:15.797642 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.797486 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istiod-ca-cert" pod "db7b6603-3dfd-4f20-a362-da44b2fb9da9" (UID: "db7b6603-3dfd-4f20-a362-da44b2fb9da9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:15.797772 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.797573 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istiod-ca-cert" pod "07fab6bc-5ee9-4724-aa10-dcf0df1af13b" (UID: "07fab6bc-5ee9-4724-aa10-dcf0df1af13b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:15.797772 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.797726 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-workload-socket" pod "630b8106-55a0-46e3-9de9-85898b01255d" (UID: "630b8106-55a0-46e3-9de9-85898b01255d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:15.797772 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.797746 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-credential-socket" pod "630b8106-55a0-46e3-9de9-85898b01255d" (UID: "630b8106-55a0-46e3-9de9-85898b01255d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:15.797943 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.797840 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/630b8106-55a0-46e3-9de9-85898b01255d-istiod-ca-cert" pod "630b8106-55a0-46e3-9de9-85898b01255d" (UID: "630b8106-55a0-46e3-9de9-85898b01255d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:15.797998 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.797944 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-workload-socket" pod "db7b6603-3dfd-4f20-a362-da44b2fb9da9" (UID: "db7b6603-3dfd-4f20-a362-da44b2fb9da9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:15.798123 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.798106 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-workload-certs" pod "07fab6bc-5ee9-4724-aa10-dcf0df1af13b" (UID: "07fab6bc-5ee9-4724-aa10-dcf0df1af13b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:15.798209 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.798122 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-data" pod "db7b6603-3dfd-4f20-a362-da44b2fb9da9" (UID: "db7b6603-3dfd-4f20-a362-da44b2fb9da9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:15.798789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.798328 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-workload-certs" pod "630b8106-55a0-46e3-9de9-85898b01255d" (UID: "630b8106-55a0-46e3-9de9-85898b01255d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:15.798789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.798498 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-credential-socket" pod "db7b6603-3dfd-4f20-a362-da44b2fb9da9" (UID: "db7b6603-3dfd-4f20-a362-da44b2fb9da9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:15.798789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.798527 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-data" pod "07fab6bc-5ee9-4724-aa10-dcf0df1af13b" (UID: "07fab6bc-5ee9-4724-aa10-dcf0df1af13b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:15.799309 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.799282 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-credential-socket" pod "07fab6bc-5ee9-4724-aa10-dcf0df1af13b" (UID: "07fab6bc-5ee9-4724-aa10-dcf0df1af13b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:15.800724 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.800695 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-podinfo" pod "db7b6603-3dfd-4f20-a362-da44b2fb9da9" (UID: "db7b6603-3dfd-4f20-a362-da44b2fb9da9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:49:15.803138 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.803114 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-podinfo" pod "07fab6bc-5ee9-4724-aa10-dcf0df1af13b" (UID: "07fab6bc-5ee9-4724-aa10-dcf0df1af13b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:49:15.803226 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.803178 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-kube-api-access-b8rpn" pod "07fab6bc-5ee9-4724-aa10-dcf0df1af13b" (UID: "07fab6bc-5ee9-4724-aa10-dcf0df1af13b"). InnerVolumeSpecName "kube-api-access-b8rpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:15.803226 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.803186 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-envoy" pod "db7b6603-3dfd-4f20-a362-da44b2fb9da9" (UID: "db7b6603-3dfd-4f20-a362-da44b2fb9da9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:15.804566 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.804538 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630b8106-55a0-46e3-9de9-85898b01255d-istio-token" pod "630b8106-55a0-46e3-9de9-85898b01255d" (UID: "630b8106-55a0-46e3-9de9-85898b01255d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:15.804658 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.804575 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-envoy" pod "07fab6bc-5ee9-4724-aa10-dcf0df1af13b" (UID: "07fab6bc-5ee9-4724-aa10-dcf0df1af13b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:15.805420 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.805384 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/630b8106-55a0-46e3-9de9-85898b01255d-istio-podinfo" pod "630b8106-55a0-46e3-9de9-85898b01255d" (UID: "630b8106-55a0-46e3-9de9-85898b01255d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:49:15.806271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.806211 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-token" pod "db7b6603-3dfd-4f20-a362-da44b2fb9da9" (UID: "db7b6603-3dfd-4f20-a362-da44b2fb9da9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:15.806271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.806244 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630b8106-55a0-46e3-9de9-85898b01255d-kube-api-access-r9xx2" pod "630b8106-55a0-46e3-9de9-85898b01255d" (UID: "630b8106-55a0-46e3-9de9-85898b01255d"). InnerVolumeSpecName "kube-api-access-r9xx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:15.806529 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.806496 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-istio-envoy" pod "630b8106-55a0-46e3-9de9-85898b01255d" (UID: "630b8106-55a0-46e3-9de9-85898b01255d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:15.806682 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.806660 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-token" pod "07fab6bc-5ee9-4724-aa10-dcf0df1af13b" (UID: "07fab6bc-5ee9-4724-aa10-dcf0df1af13b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:15.808656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.808629 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db7b6603-3dfd-4f20-a362-da44b2fb9da9-kube-api-access-nxq25" pod "db7b6603-3dfd-4f20-a362-da44b2fb9da9" (UID: "db7b6603-3dfd-4f20-a362-da44b2fb9da9"). InnerVolumeSpecName "kube-api-access-nxq25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:15.897965 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.897935 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.897965 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.897960 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.897965 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.897972 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.898419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.897981 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.898419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.897989 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.898419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.897999 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.898419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.898009 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.898419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.898017 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.898419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.898026 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.898419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.898034 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9xx2\" (UniqueName: \"kubernetes.io/projected/630b8106-55a0-46e3-9de9-85898b01255d-kube-api-access-r9xx2\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.898419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.898044 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.898419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.898052 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.898419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.898060 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8rpn\" (UniqueName: \"kubernetes.io/projected/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-kube-api-access-b8rpn\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.898419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.898068 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.898419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.898077 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.898419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.898084 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630b8106-55a0-46e3-9de9-85898b01255d-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.898419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.898092 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630b8106-55a0-46e3-9de9-85898b01255d-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.898419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.898101 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.898419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.898109 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.898419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.898117 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.898419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.898128 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07fab6bc-5ee9-4724-aa10-dcf0df1af13b-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.898419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.898137 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7b6603-3dfd-4f20-a362-da44b2fb9da9-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.898419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.898145 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7b6603-3dfd-4f20-a362-da44b2fb9da9-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.898419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.898154 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxq25\" (UniqueName: \"kubernetes.io/projected/db7b6603-3dfd-4f20-a362-da44b2fb9da9-kube-api-access-nxq25\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.898419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.898164 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630b8106-55a0-46e3-9de9-85898b01255d-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.898419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:15.898173 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630b8106-55a0-46e3-9de9-85898b01255d-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:16.202006 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.201961 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:16.492297 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.492216 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" Oct 30 21:49:16.492297 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.492224 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc" event={"ID":"07fab6bc-5ee9-4724-aa10-dcf0df1af13b","Type":"ContainerDied","Data":"4392d179f842f247143603d2e37e0a99d3d4812aef88cd4dd53863d12c29c5a0"} Oct 30 21:49:16.492297 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.492283 5801 scope.go:122] "RemoveContainer" containerID="a19a452c99d9201e48336b7e0e807f5096f517fe29e1f9c6137146695c1bcaee" Oct 30 21:49:16.493907 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.493886 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" Oct 30 21:49:16.494043 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.493886 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r" event={"ID":"db7b6603-3dfd-4f20-a362-da44b2fb9da9","Type":"ContainerDied","Data":"d6d154e7a275586ca06112a3271387081e56d14ddf8a6f2f29dcd3374c761b4c"} Oct 30 21:49:16.495343 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.495325 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" Oct 30 21:49:16.495448 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.495347 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89" event={"ID":"630b8106-55a0-46e3-9de9-85898b01255d","Type":"ContainerDied","Data":"917b8c735de9eca2151b9ac753a32f0788a8de6b2353bcc9bc816d4bfd7ccc73"} Oct 30 21:49:16.497334 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.497307 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63561e84-hqgzs" event={"ID":"0d648b4b-c946-4b26-b77b-a3837e5b2135","Type":"ContainerStarted","Data":"63b7151335959dd223ebdf01c046ef23652805a21dae786f756d66e9c7df2361"} Oct 30 21:49:16.498103 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.497358 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63561e84-hqgzs" event={"ID":"0d648b4b-c946-4b26-b77b-a3837e5b2135","Type":"ContainerStarted","Data":"ba015496a81d3db582dbcb33bc7b42790aa0bc55a4e60938c29915e4b3b44f17"} Oct 30 21:49:16.499963 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.499936 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ffa292-8b8gj" event={"ID":"eecf577b-fd05-4675-b050-5cf510babaa4","Type":"ContainerStarted","Data":"fdfa9918702e0234bc088474bdd102a7563ca0610d4695ea486ec4528c0598a2"} Oct 30 21:49:16.500086 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.499988 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ffa292-8b8gj" event={"ID":"eecf577b-fd05-4675-b050-5cf510babaa4","Type":"ContainerStarted","Data":"ff6465c36003428bae3ae69cacab08f72df2c177d929759f8b07ffa94097793c"} Oct 30 21:49:16.507567 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.507529 5801 scope.go:122] "RemoveContainer" containerID="d9e6af01d26650702e74d0f2291aa5472f8bc11ab3426247d2970ea56202f1cb" Oct 30 21:49:16.520580 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.520547 5801 scope.go:122] "RemoveContainer" containerID="17f889ac80e2685467e87c968ee179f97b0ab247d9498a53dce650ea89780c92" Oct 30 21:49:16.529806 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.529781 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc"] Oct 30 21:49:16.534408 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.534381 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-tkzlc"] Oct 30 21:49:16.549803 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.549737 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63561e84-hqgzs" podStartSLOduration=2.549724146 podStartE2EDuration="2.549724146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:16.546359001 +0000 UTC m=+1225.142175662" watchObservedRunningTime="2026-10-30 21:49:16.549724146 +0000 UTC m=+1225.145540849" Oct 30 21:49:16.570475 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.570418 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ffa292-8b8gj" podStartSLOduration=2.570401703 podStartE2EDuration="2.570401703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:16.564474241 +0000 UTC m=+1225.160290952" watchObservedRunningTime="2026-10-30 21:49:16.570401703 +0000 UTC m=+1225.166218365" Oct 30 21:49:16.587575 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.587535 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r"] Oct 30 21:49:16.595636 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.595301 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-m844r"] Oct 30 21:49:16.606488 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.606407 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89"] Oct 30 21:49:16.611215 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.611189 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-pkf89"] Oct 30 21:49:16.778026 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.777964 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb" Oct 30 21:49:16.911313 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.911277 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9-maas-api-xks-ca\") pod \"9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9\" (UID: \"9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9\") " Oct 30 21:49:16.911701 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.911341 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9-maas-api-openshift-ca\") pod \"9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9\" (UID: \"9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9\") " Oct 30 21:49:16.911701 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.911595 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9-kube-api-access-qwpvq\" (UniqueName: \"kubernetes.io/projected/9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9-kube-api-access-qwpvq\") pod \"9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9\" (UID: \"9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9\") " Oct 30 21:49:16.911701 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.911656 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9-maas-api-openshift-ca" pod "9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9" (UID: "9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:16.912257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.912230 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:16.913352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.913332 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9-maas-api-xks-ca" pod "9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9" (UID: "9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:16.913606 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:16.913586 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9-kube-api-access-qwpvq" pod "9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9" (UID: "9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9"). InnerVolumeSpecName "kube-api-access-qwpvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:17.013995 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:17.013962 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:17.013995 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:17.013994 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwpvq\" (UniqueName: \"kubernetes.io/projected/9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9-kube-api-access-qwpvq\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:17.116430 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:17.116388 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:17.510978 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:17.510888 5801 generic.go:356] "Generic (PLEG): container finished" podID="0d648b4b-c946-4b26-b77b-a3837e5b2135" containerID="63b7151335959dd223ebdf01c046ef23652805a21dae786f756d66e9c7df2361" exitCode=0 Oct 30 21:49:17.511138 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:17.511026 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63561e84-hqgzs" event={"ID":"0d648b4b-c946-4b26-b77b-a3837e5b2135","Type":"ContainerDied","Data":"63b7151335959dd223ebdf01c046ef23652805a21dae786f756d66e9c7df2361"} Oct 30 21:49:17.512460 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:17.512440 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb" Oct 30 21:49:17.512609 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:17.512456 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb" event={"ID":"9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9","Type":"ContainerDied","Data":"08f2371370490d348866e556ef1142e111e13f184b512967cf6a63f49acda952"} Oct 30 21:49:17.512609 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:17.512498 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08f2371370490d348866e556ef1142e111e13f184b512967cf6a63f49acda952" Oct 30 21:49:17.517450 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:17.514666 5801 generic.go:356] "Generic (PLEG): container finished" podID="eecf577b-fd05-4675-b050-5cf510babaa4" containerID="fdfa9918702e0234bc088474bdd102a7563ca0610d4695ea486ec4528c0598a2" exitCode=0 Oct 30 21:49:17.517450 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:17.514906 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ffa292-8b8gj" event={"ID":"eecf577b-fd05-4675-b050-5cf510babaa4","Type":"ContainerDied","Data":"fdfa9918702e0234bc088474bdd102a7563ca0610d4695ea486ec4528c0598a2"} Oct 30 21:49:17.531117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:17.531079 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07fab6bc-5ee9-4724-aa10-dcf0df1af13b" path="/var/lib/kubelet/pods/07fab6bc-5ee9-4724-aa10-dcf0df1af13b/volumes" Oct 30 21:49:17.531902 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:17.531881 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="630b8106-55a0-46e3-9de9-85898b01255d" path="/var/lib/kubelet/pods/630b8106-55a0-46e3-9de9-85898b01255d/volumes" Oct 30 21:49:17.532534 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:17.532515 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db7b6603-3dfd-4f20-a362-da44b2fb9da9" path="/var/lib/kubelet/pods/db7b6603-3dfd-4f20-a362-da44b2fb9da9/volumes" Oct 30 21:49:17.826926 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:17.826885 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:18.056942 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:18.056905 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:18.578786 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:18.578731 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:18.729197 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:18.729050 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:18.842797 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:18.841331 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:18.896547 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:18.895007 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ffa292-8b8gj" Oct 30 21:49:18.897686 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:18.897656 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63561e84-hqgzs" Oct 30 21:49:18.899962 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:18.899937 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t"] Oct 30 21:49:18.905836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:18.905248 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08aafeac-9rs5t"] Oct 30 21:49:18.983576 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:18.983532 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d648b4b-c946-4b26-b77b-a3837e5b2135-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d648b4b-c946-4b26-b77b-a3837e5b2135-maas-api-xks-ca\") pod \"0d648b4b-c946-4b26-b77b-a3837e5b2135\" (UID: \"0d648b4b-c946-4b26-b77b-a3837e5b2135\") " Oct 30 21:49:18.983837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:18.983782 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d648b4b-c946-4b26-b77b-a3837e5b2135-kube-api-access-5rjz2\" (UniqueName: \"kubernetes.io/projected/0d648b4b-c946-4b26-b77b-a3837e5b2135-kube-api-access-5rjz2\") pod \"0d648b4b-c946-4b26-b77b-a3837e5b2135\" (UID: \"0d648b4b-c946-4b26-b77b-a3837e5b2135\") " Oct 30 21:49:18.983920 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:18.983852 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d648b4b-c946-4b26-b77b-a3837e5b2135-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d648b4b-c946-4b26-b77b-a3837e5b2135-maas-api-openshift-ca\") pod \"0d648b4b-c946-4b26-b77b-a3837e5b2135\" (UID: \"0d648b4b-c946-4b26-b77b-a3837e5b2135\") " Oct 30 21:49:18.983991 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:18.983940 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eecf577b-fd05-4675-b050-5cf510babaa4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eecf577b-fd05-4675-b050-5cf510babaa4-maas-api-openshift-ca\") pod \"eecf577b-fd05-4675-b050-5cf510babaa4\" (UID: \"eecf577b-fd05-4675-b050-5cf510babaa4\") " Oct 30 21:49:18.983991 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:18.983980 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eecf577b-fd05-4675-b050-5cf510babaa4-kube-api-access-xts9g\" (UniqueName: \"kubernetes.io/projected/eecf577b-fd05-4675-b050-5cf510babaa4-kube-api-access-xts9g\") pod \"eecf577b-fd05-4675-b050-5cf510babaa4\" (UID: \"eecf577b-fd05-4675-b050-5cf510babaa4\") " Oct 30 21:49:18.984077 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:18.984059 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eecf577b-fd05-4675-b050-5cf510babaa4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eecf577b-fd05-4675-b050-5cf510babaa4-maas-api-xks-ca\") pod \"eecf577b-fd05-4675-b050-5cf510babaa4\" (UID: \"eecf577b-fd05-4675-b050-5cf510babaa4\") " Oct 30 21:49:18.984488 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:18.984452 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d648b4b-c946-4b26-b77b-a3837e5b2135-maas-api-openshift-ca" pod "0d648b4b-c946-4b26-b77b-a3837e5b2135" (UID: "0d648b4b-c946-4b26-b77b-a3837e5b2135"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:18.984488 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:18.984462 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eecf577b-fd05-4675-b050-5cf510babaa4-maas-api-openshift-ca" pod "eecf577b-fd05-4675-b050-5cf510babaa4" (UID: "eecf577b-fd05-4675-b050-5cf510babaa4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:18.984816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:18.984796 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d648b4b-c946-4b26-b77b-a3837e5b2135-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:18.984908 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:18.984820 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eecf577b-fd05-4675-b050-5cf510babaa4-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:18.987287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:18.987260 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eecf577b-fd05-4675-b050-5cf510babaa4-kube-api-access-xts9g" pod "eecf577b-fd05-4675-b050-5cf510babaa4" (UID: "eecf577b-fd05-4675-b050-5cf510babaa4"). InnerVolumeSpecName "kube-api-access-xts9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:18.987388 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:18.987305 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d648b4b-c946-4b26-b77b-a3837e5b2135-kube-api-access-5rjz2" pod "0d648b4b-c946-4b26-b77b-a3837e5b2135" (UID: "0d648b4b-c946-4b26-b77b-a3837e5b2135"). InnerVolumeSpecName "kube-api-access-5rjz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:18.987607 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:18.987574 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eecf577b-fd05-4675-b050-5cf510babaa4-maas-api-xks-ca" pod "eecf577b-fd05-4675-b050-5cf510babaa4" (UID: "eecf577b-fd05-4675-b050-5cf510babaa4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:18.987607 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:18.987591 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d648b4b-c946-4b26-b77b-a3837e5b2135-maas-api-xks-ca" pod "0d648b4b-c946-4b26-b77b-a3837e5b2135" (UID: "0d648b4b-c946-4b26-b77b-a3837e5b2135"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:19.059236 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:19.059182 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:19.087257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:19.087150 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eecf577b-fd05-4675-b050-5cf510babaa4-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:19.087257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:19.087189 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d648b4b-c946-4b26-b77b-a3837e5b2135-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:19.087257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:19.087211 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rjz2\" (UniqueName: \"kubernetes.io/projected/0d648b4b-c946-4b26-b77b-a3837e5b2135-kube-api-access-5rjz2\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:19.087257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:19.087228 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xts9g\" (UniqueName: \"kubernetes.io/projected/eecf577b-fd05-4675-b050-5cf510babaa4-kube-api-access-xts9g\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:19.430808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:19.430668 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:19.532040 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:19.532002 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3ef08c1-b100-409a-ab98-f36965c0c826" path="/var/lib/kubelet/pods/b3ef08c1-b100-409a-ab98-f36965c0c826/volumes" Oct 30 21:49:19.554740 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:19.554709 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63561e84-hqgzs" event={"ID":"0d648b4b-c946-4b26-b77b-a3837e5b2135","Type":"ContainerDied","Data":"ba015496a81d3db582dbcb33bc7b42790aa0bc55a4e60938c29915e4b3b44f17"} Oct 30 21:49:19.554740 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:19.554728 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63561e84-hqgzs" Oct 30 21:49:19.554740 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:19.554773 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba015496a81d3db582dbcb33bc7b42790aa0bc55a4e60938c29915e4b3b44f17" Oct 30 21:49:19.556549 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:19.556531 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ffa292-8b8gj" Oct 30 21:49:19.556661 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:19.556572 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ffa292-8b8gj" event={"ID":"eecf577b-fd05-4675-b050-5cf510babaa4","Type":"ContainerDied","Data":"ff6465c36003428bae3ae69cacab08f72df2c177d929759f8b07ffa94097793c"} Oct 30 21:49:19.556661 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:19.556617 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff6465c36003428bae3ae69cacab08f72df2c177d929759f8b07ffa94097793c" Oct 30 21:49:19.863911 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:19.863872 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:20.558450 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:20.558403 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:21.315730 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:21.315692 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:22.138472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:22.138435 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:22.826307 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:22.826264 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:23.032291 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:23.032251 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:23.141957 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:23.141846 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr"] Oct 30 21:49:23.151364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:23.151307 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cd3c4c-5ngmr"] Oct 30 21:49:23.180312 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:23.180279 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:23.436842 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:23.436259 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:23.527628 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:23.527597 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbc85c6d-b1d2-405d-b3cf-de42924b33e3" path="/var/lib/kubelet/pods/bbc85c6d-b1d2-405d-b3cf-de42924b33e3/volumes" Oct 30 21:49:23.538951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:23.538922 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:23.697086 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:23.696988 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:23.973257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:23.973147 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79" podUID="c7b2ea58-b940-4434-8b89-c3d7c0bc3cba" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:49:24.269371 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:24.269260 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:24.478046 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:24.478006 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:24.731916 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:24.731882 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42"] Oct 30 21:49:24.732199 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:24.732168 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" podUID="b8cc8b5b-1443-40da-a41d-745fcba8e474" containerName="istio-proxy" containerID="cri-o://65f4a6712c38dd9c3273bed541a9589206a7619bc98a1b557cf337bdb1107e9f" gracePeriod=30 Oct 30 21:49:24.736506 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:24.736482 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2"] Oct 30 21:49:24.736668 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:24.736649 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" podUID="9e3895a8-b605-453e-8b79-93b54f506dfb" containerName="istio-proxy" containerID="cri-o://6b0b8f133c1075b48c33ad18505557780d189c2034c8e5824b37eafbbe385ebc" gracePeriod=30 Oct 30 21:49:24.840345 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:24.840305 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:25.276227 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:25.276190 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:26.170453 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:26.170413 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:26.258316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:26.258288 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79" Oct 30 21:49:26.382703 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:26.382668 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b2ea58-b940-4434-8b89-c3d7c0bc3cba-kube-api-access-qhjp4\" (UniqueName: \"kubernetes.io/projected/c7b2ea58-b940-4434-8b89-c3d7c0bc3cba-kube-api-access-qhjp4\") pod \"c7b2ea58-b940-4434-8b89-c3d7c0bc3cba\" (UID: \"c7b2ea58-b940-4434-8b89-c3d7c0bc3cba\") " Oct 30 21:49:26.383140 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:26.382902 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b2ea58-b940-4434-8b89-c3d7c0bc3cba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b2ea58-b940-4434-8b89-c3d7c0bc3cba-plugins-config-volume\") pod \"c7b2ea58-b940-4434-8b89-c3d7c0bc3cba\" (UID: \"c7b2ea58-b940-4434-8b89-c3d7c0bc3cba\") " Oct 30 21:49:26.383316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:26.383293 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b2ea58-b940-4434-8b89-c3d7c0bc3cba-plugins-config-volume" pod "c7b2ea58-b940-4434-8b89-c3d7c0bc3cba" (UID: "c7b2ea58-b940-4434-8b89-c3d7c0bc3cba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:26.383460 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:26.383442 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b2ea58-b940-4434-8b89-c3d7c0bc3cba-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:26.384855 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:26.384832 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b2ea58-b940-4434-8b89-c3d7c0bc3cba-kube-api-access-qhjp4" pod "c7b2ea58-b940-4434-8b89-c3d7c0bc3cba" (UID: "c7b2ea58-b940-4434-8b89-c3d7c0bc3cba"). InnerVolumeSpecName "kube-api-access-qhjp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:26.485253 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:26.485168 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhjp4\" (UniqueName: \"kubernetes.io/projected/c7b2ea58-b940-4434-8b89-c3d7c0bc3cba-kube-api-access-qhjp4\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:26.617119 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:26.617083 5801 generic.go:356] "Generic (PLEG): container finished" podID="c7b2ea58-b940-4434-8b89-c3d7c0bc3cba" containerID="12de89c06f58497c1090ae71ea18c1f56245e65d5e1ab8ab7266403227676ca4" exitCode=0 Oct 30 21:49:26.617317 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:26.617131 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79" Oct 30 21:49:26.617317 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:26.617164 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79" event={"ID":"c7b2ea58-b940-4434-8b89-c3d7c0bc3cba","Type":"ContainerDied","Data":"12de89c06f58497c1090ae71ea18c1f56245e65d5e1ab8ab7266403227676ca4"} Oct 30 21:49:26.617317 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:26.617217 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79" event={"ID":"c7b2ea58-b940-4434-8b89-c3d7c0bc3cba","Type":"ContainerDied","Data":"bf97457de0ac059e9debfd7ca7550726c96c066b3c547fdb708830bd400f2366"} Oct 30 21:49:26.617317 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:26.617241 5801 scope.go:122] "RemoveContainer" containerID="12de89c06f58497c1090ae71ea18c1f56245e65d5e1ab8ab7266403227676ca4" Oct 30 21:49:26.628583 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:26.628564 5801 scope.go:122] "RemoveContainer" containerID="12de89c06f58497c1090ae71ea18c1f56245e65d5e1ab8ab7266403227676ca4" Oct 30 21:49:26.628866 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:49:26.628843 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12de89c06f58497c1090ae71ea18c1f56245e65d5e1ab8ab7266403227676ca4\": container with ID starting with 12de89c06f58497c1090ae71ea18c1f56245e65d5e1ab8ab7266403227676ca4 not found: ID does not exist" containerID="12de89c06f58497c1090ae71ea18c1f56245e65d5e1ab8ab7266403227676ca4" Oct 30 21:49:26.628940 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:26.628879 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12de89c06f58497c1090ae71ea18c1f56245e65d5e1ab8ab7266403227676ca4"} err="failed to get container status \"12de89c06f58497c1090ae71ea18c1f56245e65d5e1ab8ab7266403227676ca4\": rpc error: code = NotFound desc = could not find container \"12de89c06f58497c1090ae71ea18c1f56245e65d5e1ab8ab7266403227676ca4\": container with ID starting with 12de89c06f58497c1090ae71ea18c1f56245e65d5e1ab8ab7266403227676ca4 not found: ID does not exist" Oct 30 21:49:26.643382 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:26.643354 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79"] Oct 30 21:49:26.647961 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:26.647942 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7dfff0-7d5756465-stk79"] Oct 30 21:49:27.135671 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:27.135635 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:27.526551 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:27.526467 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b2ea58-b940-4434-8b89-c3d7c0bc3cba" path="/var/lib/kubelet/pods/c7b2ea58-b940-4434-8b89-c3d7c0bc3cba/volumes" Oct 30 21:49:27.863308 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:27.863274 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:27.930920 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:27.930860 5801 patch_prober.go:29] interesting pod/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:49:27.931160 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:27.930940 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" podUID="b8cc8b5b-1443-40da-a41d-745fcba8e474" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:49:28.527744 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:28.527698 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:29.131782 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.130956 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d631ede9-h847l"] Oct 30 21:49:29.133951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.132068 5801 cpu_manager.go:414] "containerMap: removing container" podUID="db7b6603-3dfd-4f20-a362-da44b2fb9da9" containerName="istio-proxy" Oct 30 21:49:29.133951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.132093 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db7b6603-3dfd-4f20-a362-da44b2fb9da9" containerName="istio-proxy" Oct 30 21:49:29.133951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.132111 5801 cpu_manager.go:414] "containerMap: removing container" podUID="07fab6bc-5ee9-4724-aa10-dcf0df1af13b" containerName="istio-proxy" Oct 30 21:49:29.133951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.132119 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07fab6bc-5ee9-4724-aa10-dcf0df1af13b" containerName="istio-proxy" Oct 30 21:49:29.133951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.132138 5801 cpu_manager.go:414] "containerMap: removing container" podUID="0d648b4b-c946-4b26-b77b-a3837e5b2135" containerName="revoke-keys" Oct 30 21:49:29.133951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.132146 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d648b4b-c946-4b26-b77b-a3837e5b2135" containerName="revoke-keys" Oct 30 21:49:29.133951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.132169 5801 cpu_manager.go:414] "containerMap: removing container" podUID="630b8106-55a0-46e3-9de9-85898b01255d" containerName="istio-proxy" Oct 30 21:49:29.133951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.132176 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="630b8106-55a0-46e3-9de9-85898b01255d" containerName="istio-proxy" Oct 30 21:49:29.133951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.132186 5801 cpu_manager.go:414] "containerMap: removing container" podUID="9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9" containerName="revoke-keys" Oct 30 21:49:29.133951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.132194 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9" containerName="revoke-keys" Oct 30 21:49:29.133951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.132205 5801 cpu_manager.go:414] "containerMap: removing container" podUID="eecf577b-fd05-4675-b050-5cf510babaa4" containerName="revoke-keys" Oct 30 21:49:29.133951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.132212 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eecf577b-fd05-4675-b050-5cf510babaa4" containerName="revoke-keys" Oct 30 21:49:29.133951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.132258 5801 cpu_manager.go:414] "containerMap: removing container" podUID="c7b2ea58-b940-4434-8b89-c3d7c0bc3cba" containerName="payload-processing" Oct 30 21:49:29.133951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.132265 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b2ea58-b940-4434-8b89-c3d7c0bc3cba" containerName="payload-processing" Oct 30 21:49:29.133951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.132441 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b2ea58-b940-4434-8b89-c3d7c0bc3cba" containerName="payload-processing" Oct 30 21:49:29.133951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.132461 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="db7b6603-3dfd-4f20-a362-da44b2fb9da9" containerName="istio-proxy" Oct 30 21:49:29.133951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.132472 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="07fab6bc-5ee9-4724-aa10-dcf0df1af13b" containerName="istio-proxy" Oct 30 21:49:29.133951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.132483 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d648b4b-c946-4b26-b77b-a3837e5b2135" containerName="revoke-keys" Oct 30 21:49:29.133951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.132495 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="630b8106-55a0-46e3-9de9-85898b01255d" containerName="istio-proxy" Oct 30 21:49:29.133951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.132505 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9" containerName="revoke-keys" Oct 30 21:49:29.133951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.132516 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="eecf577b-fd05-4675-b050-5cf510babaa4" containerName="revoke-keys" Oct 30 21:49:29.145556 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.145521 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d631ede9-h847l" Oct 30 21:49:29.151146 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.151001 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:49:29.151146 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.151094 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:49:29.152817 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.152721 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:49:29.152817 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.152792 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:49:29.158053 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.158033 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc4da15-cqdfn"] Oct 30 21:49:29.163608 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.163122 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc4da15-cqdfn" Oct 30 21:49:29.191272 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.191242 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d631ede9-h847l"] Oct 30 21:49:29.203822 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.202865 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc4da15-cqdfn"] Oct 30 21:49:29.260956 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.260915 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8425a8c1-cff8-42cd-af91-c6b3b2f8d222-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d631ede9-h847l\" (UID: \"8425a8c1-cff8-42cd-af91-c6b3b2f8d222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d631ede9-h847l" Oct 30 21:49:29.262423 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.262377 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8425a8c1-cff8-42cd-af91-c6b3b2f8d222-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d631ede9-h847l\" (UID: \"8425a8c1-cff8-42cd-af91-c6b3b2f8d222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d631ede9-h847l" Oct 30 21:49:29.262688 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.262664 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8sqs\" (UniqueName: \"kubernetes.io/projected/8425a8c1-cff8-42cd-af91-c6b3b2f8d222-kube-api-access-f8sqs\") pod \"maas-api-revoke-sub-d631ede9-h847l\" (UID: \"8425a8c1-cff8-42cd-af91-c6b3b2f8d222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d631ede9-h847l" Oct 30 21:49:29.365222 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.365189 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaace599-087f-4a86-91a6-2abd2a65cfaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebc4da15-cqdfn\" (UID: \"aaace599-087f-4a86-91a6-2abd2a65cfaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc4da15-cqdfn" Oct 30 21:49:29.365415 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.365363 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8425a8c1-cff8-42cd-af91-c6b3b2f8d222-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d631ede9-h847l\" (UID: \"8425a8c1-cff8-42cd-af91-c6b3b2f8d222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d631ede9-h847l" Oct 30 21:49:29.365465 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.365429 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j46k\" (UniqueName: \"kubernetes.io/projected/aaace599-087f-4a86-91a6-2abd2a65cfaa-kube-api-access-7j46k\") pod \"maas-api-revoke-sub-ebc4da15-cqdfn\" (UID: \"aaace599-087f-4a86-91a6-2abd2a65cfaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc4da15-cqdfn" Oct 30 21:49:29.365510 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.365480 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8sqs\" (UniqueName: \"kubernetes.io/projected/8425a8c1-cff8-42cd-af91-c6b3b2f8d222-kube-api-access-f8sqs\") pod \"maas-api-revoke-sub-d631ede9-h847l\" (UID: \"8425a8c1-cff8-42cd-af91-c6b3b2f8d222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d631ede9-h847l" Oct 30 21:49:29.365549 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.365534 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaace599-087f-4a86-91a6-2abd2a65cfaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebc4da15-cqdfn\" (UID: \"aaace599-087f-4a86-91a6-2abd2a65cfaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc4da15-cqdfn" Oct 30 21:49:29.365796 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.365775 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8425a8c1-cff8-42cd-af91-c6b3b2f8d222-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d631ede9-h847l\" (UID: \"8425a8c1-cff8-42cd-af91-c6b3b2f8d222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d631ede9-h847l" Oct 30 21:49:29.366110 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.366086 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8425a8c1-cff8-42cd-af91-c6b3b2f8d222-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d631ede9-h847l\" (UID: \"8425a8c1-cff8-42cd-af91-c6b3b2f8d222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d631ede9-h847l" Oct 30 21:49:29.368368 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.368313 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8425a8c1-cff8-42cd-af91-c6b3b2f8d222-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d631ede9-h847l\" (UID: \"8425a8c1-cff8-42cd-af91-c6b3b2f8d222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d631ede9-h847l" Oct 30 21:49:29.379463 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.379437 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8sqs\" (UniqueName: \"kubernetes.io/projected/8425a8c1-cff8-42cd-af91-c6b3b2f8d222-kube-api-access-f8sqs\") pod \"maas-api-revoke-sub-d631ede9-h847l\" (UID: \"8425a8c1-cff8-42cd-af91-c6b3b2f8d222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d631ede9-h847l" Oct 30 21:49:29.479870 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.474979 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaace599-087f-4a86-91a6-2abd2a65cfaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebc4da15-cqdfn\" (UID: \"aaace599-087f-4a86-91a6-2abd2a65cfaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc4da15-cqdfn" Oct 30 21:49:29.479870 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.475146 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j46k\" (UniqueName: \"kubernetes.io/projected/aaace599-087f-4a86-91a6-2abd2a65cfaa-kube-api-access-7j46k\") pod \"maas-api-revoke-sub-ebc4da15-cqdfn\" (UID: \"aaace599-087f-4a86-91a6-2abd2a65cfaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc4da15-cqdfn" Oct 30 21:49:29.479870 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.475209 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaace599-087f-4a86-91a6-2abd2a65cfaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebc4da15-cqdfn\" (UID: \"aaace599-087f-4a86-91a6-2abd2a65cfaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc4da15-cqdfn" Oct 30 21:49:29.479870 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.476248 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaace599-087f-4a86-91a6-2abd2a65cfaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebc4da15-cqdfn\" (UID: \"aaace599-087f-4a86-91a6-2abd2a65cfaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc4da15-cqdfn" Oct 30 21:49:29.482538 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.482405 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaace599-087f-4a86-91a6-2abd2a65cfaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebc4da15-cqdfn\" (UID: \"aaace599-087f-4a86-91a6-2abd2a65cfaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc4da15-cqdfn" Oct 30 21:49:29.491818 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.489006 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j46k\" (UniqueName: \"kubernetes.io/projected/aaace599-087f-4a86-91a6-2abd2a65cfaa-kube-api-access-7j46k\") pod \"maas-api-revoke-sub-ebc4da15-cqdfn\" (UID: \"aaace599-087f-4a86-91a6-2abd2a65cfaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc4da15-cqdfn" Oct 30 21:49:29.494800 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.494770 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d631ede9-h847l" Oct 30 21:49:29.497926 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.497901 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc4da15-cqdfn" Oct 30 21:49:29.513389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.513355 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:29.946186 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:29.946153 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d631ede9-h847l"] Oct 30 21:49:30.090052 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.090018 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc4da15-cqdfn"] Oct 30 21:49:30.105106 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.105067 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:30.159888 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:49:30.159858 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaace599_087f_4a86_91a6_2abd2a65cfaa.slice/crio-0fd57c7c7b533073c70d2f8489cb984fbb3319bfaf92db8e0df6b00310eae21e WatchSource:0}: Error finding container 0fd57c7c7b533073c70d2f8489cb984fbb3319bfaf92db8e0df6b00310eae21e: Status 404 returned error can't find the container with id 0fd57c7c7b533073c70d2f8489cb984fbb3319bfaf92db8e0df6b00310eae21e Oct 30 21:49:30.182584 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.182562 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:49:30.185369 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.185349 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:49:30.289468 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.289430 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:30.293703 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.293672 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-token\" (UniqueName: \"kubernetes.io/projected/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-token\") pod \"9e3895a8-b605-453e-8b79-93b54f506dfb\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " Oct 30 21:49:30.293861 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.293719 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-token\" (UniqueName: \"kubernetes.io/projected/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-token\") pod \"b8cc8b5b-1443-40da-a41d-745fcba8e474\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " Oct 30 21:49:30.293861 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.293790 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-podinfo\") pod \"b8cc8b5b-1443-40da-a41d-745fcba8e474\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " Oct 30 21:49:30.294001 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.293860 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e3895a8-b605-453e-8b79-93b54f506dfb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3895a8-b605-453e-8b79-93b54f506dfb-istiod-ca-cert\") pod \"9e3895a8-b605-453e-8b79-93b54f506dfb\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " Oct 30 21:49:30.294001 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.293890 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3895a8-b605-453e-8b79-93b54f506dfb-kube-api-access-khd78\" (UniqueName: \"kubernetes.io/projected/9e3895a8-b605-453e-8b79-93b54f506dfb-kube-api-access-khd78\") pod \"9e3895a8-b605-453e-8b79-93b54f506dfb\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " Oct 30 21:49:30.294001 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.293934 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-workload-socket\") pod \"9e3895a8-b605-453e-8b79-93b54f506dfb\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " Oct 30 21:49:30.294168 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.294023 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-envoy\") pod \"b8cc8b5b-1443-40da-a41d-745fcba8e474\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " Oct 30 21:49:30.294168 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.294110 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-data\") pod \"9e3895a8-b605-453e-8b79-93b54f506dfb\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " Oct 30 21:49:30.294168 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.294161 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-data\") pod \"b8cc8b5b-1443-40da-a41d-745fcba8e474\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " Oct 30 21:49:30.294331 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.294200 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-credential-socket\") pod \"9e3895a8-b605-453e-8b79-93b54f506dfb\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " Oct 30 21:49:30.294331 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.294240 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-workload-certs\") pod \"9e3895a8-b605-453e-8b79-93b54f506dfb\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " Oct 30 21:49:30.294331 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.294319 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-envoy\") pod \"9e3895a8-b605-453e-8b79-93b54f506dfb\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " Oct 30 21:49:30.294492 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.294365 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-workload-certs\") pod \"b8cc8b5b-1443-40da-a41d-745fcba8e474\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " Oct 30 21:49:30.294492 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.294378 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e3895a8-b605-453e-8b79-93b54f506dfb-istiod-ca-cert" pod "9e3895a8-b605-453e-8b79-93b54f506dfb" (UID: "9e3895a8-b605-453e-8b79-93b54f506dfb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:30.294492 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.294374 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-workload-socket" pod "9e3895a8-b605-453e-8b79-93b54f506dfb" (UID: "9e3895a8-b605-453e-8b79-93b54f506dfb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:30.294492 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.294466 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8cc8b5b-1443-40da-a41d-745fcba8e474-kube-api-access-d44bq\" (UniqueName: \"kubernetes.io/projected/b8cc8b5b-1443-40da-a41d-745fcba8e474-kube-api-access-d44bq\") pod \"b8cc8b5b-1443-40da-a41d-745fcba8e474\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " Oct 30 21:49:30.294718 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.294520 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-workload-socket\") pod \"b8cc8b5b-1443-40da-a41d-745fcba8e474\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " Oct 30 21:49:30.294718 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.294596 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-credential-socket" pod "9e3895a8-b605-453e-8b79-93b54f506dfb" (UID: "9e3895a8-b605-453e-8b79-93b54f506dfb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:30.294718 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.294602 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-workload-certs" pod "b8cc8b5b-1443-40da-a41d-745fcba8e474" (UID: "b8cc8b5b-1443-40da-a41d-745fcba8e474"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:30.294718 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.294630 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8cc8b5b-1443-40da-a41d-745fcba8e474-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cc8b5b-1443-40da-a41d-745fcba8e474-istiod-ca-cert\") pod \"b8cc8b5b-1443-40da-a41d-745fcba8e474\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " Oct 30 21:49:30.294718 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.294666 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-credential-socket\") pod \"b8cc8b5b-1443-40da-a41d-745fcba8e474\" (UID: \"b8cc8b5b-1443-40da-a41d-745fcba8e474\") " Oct 30 21:49:30.295027 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.294746 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-podinfo\") pod \"9e3895a8-b605-453e-8b79-93b54f506dfb\" (UID: \"9e3895a8-b605-453e-8b79-93b54f506dfb\") " Oct 30 21:49:30.295027 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.294795 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-workload-certs" pod "9e3895a8-b605-453e-8b79-93b54f506dfb" (UID: "9e3895a8-b605-453e-8b79-93b54f506dfb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:30.295027 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.294806 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-data" pod "9e3895a8-b605-453e-8b79-93b54f506dfb" (UID: "9e3895a8-b605-453e-8b79-93b54f506dfb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:30.295027 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.294890 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-workload-socket" pod "b8cc8b5b-1443-40da-a41d-745fcba8e474" (UID: "b8cc8b5b-1443-40da-a41d-745fcba8e474"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:30.295027 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.295001 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-data" pod "b8cc8b5b-1443-40da-a41d-745fcba8e474" (UID: "b8cc8b5b-1443-40da-a41d-745fcba8e474"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:30.296150 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.295359 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-credential-socket" pod "b8cc8b5b-1443-40da-a41d-745fcba8e474" (UID: "b8cc8b5b-1443-40da-a41d-745fcba8e474"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:30.296150 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.295368 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8cc8b5b-1443-40da-a41d-745fcba8e474-istiod-ca-cert" pod "b8cc8b5b-1443-40da-a41d-745fcba8e474" (UID: "b8cc8b5b-1443-40da-a41d-745fcba8e474"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:30.296150 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.296136 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3895a8-b605-453e-8b79-93b54f506dfb-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.296401 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.296180 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.296401 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.296197 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.296401 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.296212 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.296401 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.296228 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.296401 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.296261 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.296401 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.296275 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.296401 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.296289 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.296401 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.296303 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cc8b5b-1443-40da-a41d-745fcba8e474-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.296401 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.296337 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.323227 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.323197 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc"] Oct 30 21:49:30.329352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.329325 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7dfff0-j4cmc"] Oct 30 21:49:30.354625 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.354529 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3895a8-b605-453e-8b79-93b54f506dfb-kube-api-access-khd78" pod "9e3895a8-b605-453e-8b79-93b54f506dfb" (UID: "9e3895a8-b605-453e-8b79-93b54f506dfb"). InnerVolumeSpecName "kube-api-access-khd78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:30.355213 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.355183 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-token" pod "b8cc8b5b-1443-40da-a41d-745fcba8e474" (UID: "b8cc8b5b-1443-40da-a41d-745fcba8e474"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:30.355530 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.355507 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-envoy" pod "b8cc8b5b-1443-40da-a41d-745fcba8e474" (UID: "b8cc8b5b-1443-40da-a41d-745fcba8e474"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:30.355731 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.355706 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-token" pod "9e3895a8-b605-453e-8b79-93b54f506dfb" (UID: "9e3895a8-b605-453e-8b79-93b54f506dfb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:30.355853 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.355738 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-podinfo" pod "b8cc8b5b-1443-40da-a41d-745fcba8e474" (UID: "b8cc8b5b-1443-40da-a41d-745fcba8e474"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:49:30.355853 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.355745 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8cc8b5b-1443-40da-a41d-745fcba8e474-kube-api-access-d44bq" pod "b8cc8b5b-1443-40da-a41d-745fcba8e474" (UID: "b8cc8b5b-1443-40da-a41d-745fcba8e474"). InnerVolumeSpecName "kube-api-access-d44bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:30.355853 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.355810 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-podinfo" pod "9e3895a8-b605-453e-8b79-93b54f506dfb" (UID: "9e3895a8-b605-453e-8b79-93b54f506dfb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:49:30.356087 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.356066 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-envoy" pod "9e3895a8-b605-453e-8b79-93b54f506dfb" (UID: "9e3895a8-b605-453e-8b79-93b54f506dfb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:30.382821 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:49:30.382779 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:49:30.397448 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.397422 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.397448 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.397447 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d44bq\" (UniqueName: \"kubernetes.io/projected/b8cc8b5b-1443-40da-a41d-745fcba8e474-kube-api-access-d44bq\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.397584 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.397459 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.397584 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.397469 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3895a8-b605-453e-8b79-93b54f506dfb-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.397584 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.397478 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.397584 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.397487 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.397584 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.397495 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khd78\" (UniqueName: \"kubernetes.io/projected/9e3895a8-b605-453e-8b79-93b54f506dfb-kube-api-access-khd78\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.397584 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.397503 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cc8b5b-1443-40da-a41d-745fcba8e474-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.466785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.465009 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:30.734042 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.733997 5801 generic.go:356] "Generic (PLEG): container finished" podID="b8cc8b5b-1443-40da-a41d-745fcba8e474" containerID="65f4a6712c38dd9c3273bed541a9589206a7619bc98a1b557cf337bdb1107e9f" exitCode=0 Oct 30 21:49:30.734560 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.734526 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" event={"ID":"b8cc8b5b-1443-40da-a41d-745fcba8e474","Type":"ContainerDied","Data":"65f4a6712c38dd9c3273bed541a9589206a7619bc98a1b557cf337bdb1107e9f"} Oct 30 21:49:30.734785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.734745 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" event={"ID":"b8cc8b5b-1443-40da-a41d-745fcba8e474","Type":"ContainerDied","Data":"d8004bcacd9d4807d9a7f69c6fcc44ef63731f9384c38c610a7e0ff8d6c192d3"} Oct 30 21:49:30.734932 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.734917 5801 scope.go:122] "RemoveContainer" containerID="65f4a6712c38dd9c3273bed541a9589206a7619bc98a1b557cf337bdb1107e9f" Oct 30 21:49:30.735177 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.735163 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42" Oct 30 21:49:30.745675 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.745646 5801 generic.go:356] "Generic (PLEG): container finished" podID="9e3895a8-b605-453e-8b79-93b54f506dfb" containerID="6b0b8f133c1075b48c33ad18505557780d189c2034c8e5824b37eafbbe385ebc" exitCode=0 Oct 30 21:49:30.746025 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.746001 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" event={"ID":"9e3895a8-b605-453e-8b79-93b54f506dfb","Type":"ContainerDied","Data":"6b0b8f133c1075b48c33ad18505557780d189c2034c8e5824b37eafbbe385ebc"} Oct 30 21:49:30.746198 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.746177 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" event={"ID":"9e3895a8-b605-453e-8b79-93b54f506dfb","Type":"ContainerDied","Data":"ae3cf7a868203a39f8bb0ece30492cfe0835e8672bc3db16fbbdc4effad108c5"} Oct 30 21:49:30.746563 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.746543 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2" Oct 30 21:49:30.806796 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.800969 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d631ede9-h847l" event={"ID":"8425a8c1-cff8-42cd-af91-c6b3b2f8d222","Type":"ContainerStarted","Data":"a3b04c5a78be84f4b79b5245c510bbdbbf1891444537e4fb3f32b0192ffe9693"} Oct 30 21:49:30.806796 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.801041 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d631ede9-h847l" event={"ID":"8425a8c1-cff8-42cd-af91-c6b3b2f8d222","Type":"ContainerStarted","Data":"5cdd31c7d60d5f83816d839dbc531e38fa1e752f63cf2e2f71143b5ee701a3b4"} Oct 30 21:49:30.819410 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.819317 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc4da15-cqdfn" event={"ID":"aaace599-087f-4a86-91a6-2abd2a65cfaa","Type":"ContainerStarted","Data":"0fd57c7c7b533073c70d2f8489cb984fbb3319bfaf92db8e0df6b00310eae21e"} Oct 30 21:49:30.927426 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.926557 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d631ede9-h847l" podStartSLOduration=1.926538083 podStartE2EDuration="1.926538083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:30.874578422 +0000 UTC m=+1239.470395086" watchObservedRunningTime="2026-10-30 21:49:30.926538083 +0000 UTC m=+1239.522354744" Oct 30 21:49:30.935970 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.935784 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:30.950882 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.950052 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2"] Oct 30 21:49:30.958632 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:30.958605 5801 scope.go:122] "RemoveContainer" containerID="65f4a6712c38dd9c3273bed541a9589206a7619bc98a1b557cf337bdb1107e9f" Oct 30 21:49:31.009144 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:31.003277 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-k7nm2"] Oct 30 21:49:31.009144 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:49:31.006455 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f4a6712c38dd9c3273bed541a9589206a7619bc98a1b557cf337bdb1107e9f\": container with ID starting with 65f4a6712c38dd9c3273bed541a9589206a7619bc98a1b557cf337bdb1107e9f not found: ID does not exist" containerID="65f4a6712c38dd9c3273bed541a9589206a7619bc98a1b557cf337bdb1107e9f" Oct 30 21:49:31.009144 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:31.006501 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f4a6712c38dd9c3273bed541a9589206a7619bc98a1b557cf337bdb1107e9f"} err="failed to get container status \"65f4a6712c38dd9c3273bed541a9589206a7619bc98a1b557cf337bdb1107e9f\": rpc error: code = NotFound desc = could not find container \"65f4a6712c38dd9c3273bed541a9589206a7619bc98a1b557cf337bdb1107e9f\": container with ID starting with 65f4a6712c38dd9c3273bed541a9589206a7619bc98a1b557cf337bdb1107e9f not found: ID does not exist" Oct 30 21:49:31.009144 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:31.006547 5801 scope.go:122] "RemoveContainer" containerID="6b0b8f133c1075b48c33ad18505557780d189c2034c8e5824b37eafbbe385ebc" Oct 30 21:49:31.009144 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:31.008801 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42"] Oct 30 21:49:31.015529 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:31.015498 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-6rw42"] Oct 30 21:49:31.021693 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:31.021670 5801 scope.go:122] "RemoveContainer" containerID="6b0b8f133c1075b48c33ad18505557780d189c2034c8e5824b37eafbbe385ebc" Oct 30 21:49:31.022119 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:49:31.022091 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b0b8f133c1075b48c33ad18505557780d189c2034c8e5824b37eafbbe385ebc\": container with ID starting with 6b0b8f133c1075b48c33ad18505557780d189c2034c8e5824b37eafbbe385ebc not found: ID does not exist" containerID="6b0b8f133c1075b48c33ad18505557780d189c2034c8e5824b37eafbbe385ebc" Oct 30 21:49:31.022256 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:31.022133 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b0b8f133c1075b48c33ad18505557780d189c2034c8e5824b37eafbbe385ebc"} err="failed to get container status \"6b0b8f133c1075b48c33ad18505557780d189c2034c8e5824b37eafbbe385ebc\": rpc error: code = NotFound desc = could not find container \"6b0b8f133c1075b48c33ad18505557780d189c2034c8e5824b37eafbbe385ebc\": container with ID starting with 6b0b8f133c1075b48c33ad18505557780d189c2034c8e5824b37eafbbe385ebc not found: ID does not exist" Oct 30 21:49:31.571778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:31.567983 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e3895a8-b605-453e-8b79-93b54f506dfb" path="/var/lib/kubelet/pods/9e3895a8-b605-453e-8b79-93b54f506dfb/volumes" Oct 30 21:49:31.571778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:31.568767 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3aef510-0168-418b-96db-214b7da928a9" path="/var/lib/kubelet/pods/b3aef510-0168-418b-96db-214b7da928a9/volumes" Oct 30 21:49:31.571778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:31.569407 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8cc8b5b-1443-40da-a41d-745fcba8e474" path="/var/lib/kubelet/pods/b8cc8b5b-1443-40da-a41d-745fcba8e474/volumes" Oct 30 21:49:31.787682 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:31.787647 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:31.842572 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:31.842474 5801 generic.go:356] "Generic (PLEG): container finished" podID="aaace599-087f-4a86-91a6-2abd2a65cfaa" containerID="d1c893f4837ddf0a02cabde274c4175fcaf15037eed25cd89b08f10722b32524" exitCode=0 Oct 30 21:49:31.843051 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:31.843026 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc4da15-cqdfn" event={"ID":"aaace599-087f-4a86-91a6-2abd2a65cfaa","Type":"ContainerDied","Data":"d1c893f4837ddf0a02cabde274c4175fcaf15037eed25cd89b08f10722b32524"} Oct 30 21:49:31.911087 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:31.910410 5801 generic.go:356] "Generic (PLEG): container finished" podID="8425a8c1-cff8-42cd-af91-c6b3b2f8d222" containerID="a3b04c5a78be84f4b79b5245c510bbdbbf1891444537e4fb3f32b0192ffe9693" exitCode=0 Oct 30 21:49:31.911087 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:31.910467 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d631ede9-h847l" event={"ID":"8425a8c1-cff8-42cd-af91-c6b3b2f8d222","Type":"ContainerDied","Data":"a3b04c5a78be84f4b79b5245c510bbdbbf1891444537e4fb3f32b0192ffe9693"} Oct 30 21:49:32.090888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:32.090844 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:32.222831 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:32.222088 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:32.382689 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:32.382655 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:32.655775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:32.623833 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:33.299790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:33.295214 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x"] Oct 30 21:49:33.299790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:33.296332 5801 cpu_manager.go:414] "containerMap: removing container" podUID="b8cc8b5b-1443-40da-a41d-745fcba8e474" containerName="istio-proxy" Oct 30 21:49:33.299790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:33.296353 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8cc8b5b-1443-40da-a41d-745fcba8e474" containerName="istio-proxy" Oct 30 21:49:33.299790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:33.296416 5801 cpu_manager.go:414] "containerMap: removing container" podUID="9e3895a8-b605-453e-8b79-93b54f506dfb" containerName="istio-proxy" Oct 30 21:49:33.299790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:33.296425 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e3895a8-b605-453e-8b79-93b54f506dfb" containerName="istio-proxy" Oct 30 21:49:33.299790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:33.296604 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8cc8b5b-1443-40da-a41d-745fcba8e474" containerName="istio-proxy" Oct 30 21:49:33.299790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:33.296628 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e3895a8-b605-453e-8b79-93b54f506dfb" containerName="istio-proxy" Oct 30 21:49:33.429850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:33.426339 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:33.448817 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:33.434812 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x" Oct 30 21:49:33.485442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:33.462544 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76tzx\" (UniqueName: \"kubernetes.io/projected/3ba41b66-8be4-4424-a326-c7e86dc1ffcf-kube-api-access-76tzx\") pod \"maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x\" (UID: \"3ba41b66-8be4-4424-a326-c7e86dc1ffcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x" Oct 30 21:49:33.485442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:33.462714 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba41b66-8be4-4424-a326-c7e86dc1ffcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x\" (UID: \"3ba41b66-8be4-4424-a326-c7e86dc1ffcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x" Oct 30 21:49:33.485442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:33.462804 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba41b66-8be4-4424-a326-c7e86dc1ffcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x\" (UID: \"3ba41b66-8be4-4424-a326-c7e86dc1ffcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x" Oct 30 21:49:33.602777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:33.595896 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76tzx\" (UniqueName: \"kubernetes.io/projected/3ba41b66-8be4-4424-a326-c7e86dc1ffcf-kube-api-access-76tzx\") pod \"maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x\" (UID: \"3ba41b66-8be4-4424-a326-c7e86dc1ffcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x" Oct 30 21:49:33.602777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:33.596065 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba41b66-8be4-4424-a326-c7e86dc1ffcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x\" (UID: \"3ba41b66-8be4-4424-a326-c7e86dc1ffcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x" Oct 30 21:49:33.602777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:33.596134 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba41b66-8be4-4424-a326-c7e86dc1ffcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x\" (UID: \"3ba41b66-8be4-4424-a326-c7e86dc1ffcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x" Oct 30 21:49:33.602777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:33.597188 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba41b66-8be4-4424-a326-c7e86dc1ffcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x\" (UID: \"3ba41b66-8be4-4424-a326-c7e86dc1ffcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x" Oct 30 21:49:33.686612 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:33.684127 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:33.686612 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:33.685626 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:33.726383 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:33.726351 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76tzx\" (UniqueName: \"kubernetes.io/projected/3ba41b66-8be4-4424-a326-c7e86dc1ffcf-kube-api-access-76tzx\") pod \"maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x\" (UID: \"3ba41b66-8be4-4424-a326-c7e86dc1ffcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x" Oct 30 21:49:33.822406 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:33.801898 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba41b66-8be4-4424-a326-c7e86dc1ffcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x\" (UID: \"3ba41b66-8be4-4424-a326-c7e86dc1ffcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x" Oct 30 21:49:33.871834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:33.868258 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 21:49:33.913777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:33.900119 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 21:49:33.934956 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:33.929222 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 21:49:33.999778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:33.996108 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x" Oct 30 21:49:34.040674 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.016434 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x"] Oct 30 21:49:34.073325 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.070713 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9"] Oct 30 21:49:34.073325 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.072840 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:34.078977 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.078949 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.164011 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.131336 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9"] Oct 30 21:49:34.229954 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.229904 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:34.255034 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.238795 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f608f7-openshift-default-dockercfg-tjgc2\"" Oct 30 21:49:34.255481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.240468 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-data\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.255730 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.255710 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-envoy\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.255928 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.255909 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-podinfo\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.256075 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.256060 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-workload-certs\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.256190 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.256176 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-credential-socket\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.256387 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.256372 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-token\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.256530 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.256516 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lxcm\" (UniqueName: \"kubernetes.io/projected/239d3785-9a31-40ed-9e63-2d7e1265b5a2-kube-api-access-2lxcm\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.256724 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.256710 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-workload-socket\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.256904 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.256887 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istiod-ca-cert\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.257826 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.245457 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:34.294129 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.271865 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afc89e9a-t8vnz"] Oct 30 21:49:34.294129 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.293892 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afc89e9a-t8vnz"] Oct 30 21:49:34.446828 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.427139 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:34.496114 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.496074 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:34.591055 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.591020 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-token\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.599929 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.599879 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lxcm\" (UniqueName: \"kubernetes.io/projected/239d3785-9a31-40ed-9e63-2d7e1265b5a2-kube-api-access-2lxcm\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.600298 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.600255 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-workload-socket\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.604862 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.600457 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istiod-ca-cert\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.605166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.605105 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-workload-socket\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.605454 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.605412 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-data\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.605682 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.605653 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-envoy\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.605924 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.605892 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-podinfo\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.606111 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.606091 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-workload-certs\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.606286 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.606259 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-credential-socket\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.606933 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.606896 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-credential-socket\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.608560 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.608524 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-data\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.617642 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.614972 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istiod-ca-cert\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.638287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.638239 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lxcm\" (UniqueName: \"kubernetes.io/projected/239d3785-9a31-40ed-9e63-2d7e1265b5a2-kube-api-access-2lxcm\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.639008 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.638979 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-token\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.645620 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.643880 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-podinfo\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.645620 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.644150 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-workload-certs\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:34.691416 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.661874 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:34.713834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.710047 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:34.723909 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.723387 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:49:34.791087 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.736536 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:49:35.089810 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.821135 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-envoy\") pod \"e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:35.089810 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.869816 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:35.100947 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:35.100897 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:35.104041 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.910870 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:35.147100 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:34.908095 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:35.147460 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:35.147442 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:49:35.150036 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:35.147617 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:49:35.150036 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:35.147658 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:49:35.221002 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:35.199209 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:35.224413 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:35.224379 5801 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Oct 30 21:49:35.329337 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:35.284709 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:49:35.329337 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:35.320140 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:35.339527 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:35.339466 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:49:35.358390 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:35.358368 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:49:35.510772 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:35.510717 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:35.651597 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:35.651567 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="379f1ea5-497a-49c5-9356-3f402dc3d335" path="/var/lib/kubelet/pods/379f1ea5-497a-49c5-9356-3f402dc3d335/volumes" Oct 30 21:49:35.671805 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:35.652102 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:49:35.671805 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:35.652148 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x"] Oct 30 21:49:35.985777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:35.980586 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d631ede9-h847l" Oct 30 21:49:36.125780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.121775 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8425a8c1-cff8-42cd-af91-c6b3b2f8d222-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8425a8c1-cff8-42cd-af91-c6b3b2f8d222-maas-api-xks-ca\") pod \"8425a8c1-cff8-42cd-af91-c6b3b2f8d222\" (UID: \"8425a8c1-cff8-42cd-af91-c6b3b2f8d222\") " Oct 30 21:49:36.125780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.121824 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8425a8c1-cff8-42cd-af91-c6b3b2f8d222-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8425a8c1-cff8-42cd-af91-c6b3b2f8d222-maas-api-openshift-ca\") pod \"8425a8c1-cff8-42cd-af91-c6b3b2f8d222\" (UID: \"8425a8c1-cff8-42cd-af91-c6b3b2f8d222\") " Oct 30 21:49:36.125780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.122078 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8425a8c1-cff8-42cd-af91-c6b3b2f8d222-kube-api-access-f8sqs\" (UniqueName: \"kubernetes.io/projected/8425a8c1-cff8-42cd-af91-c6b3b2f8d222-kube-api-access-f8sqs\") pod \"8425a8c1-cff8-42cd-af91-c6b3b2f8d222\" (UID: \"8425a8c1-cff8-42cd-af91-c6b3b2f8d222\") " Oct 30 21:49:36.125780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.125070 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8425a8c1-cff8-42cd-af91-c6b3b2f8d222-maas-api-openshift-ca" pod "8425a8c1-cff8-42cd-af91-c6b3b2f8d222" (UID: "8425a8c1-cff8-42cd-af91-c6b3b2f8d222"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:36.141516 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.141133 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc4da15-cqdfn" Oct 30 21:49:36.142519 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.142490 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d631ede9-h847l" Oct 30 21:49:36.142642 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.142594 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d631ede9-h847l" event={"ID":"8425a8c1-cff8-42cd-af91-c6b3b2f8d222","Type":"ContainerDied","Data":"5cdd31c7d60d5f83816d839dbc531e38fa1e752f63cf2e2f71143b5ee701a3b4"} Oct 30 21:49:36.142642 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.142625 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cdd31c7d60d5f83816d839dbc531e38fa1e752f63cf2e2f71143b5ee701a3b4" Oct 30 21:49:36.144745 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.144728 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc4da15-cqdfn" Oct 30 21:49:36.144952 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.144932 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc4da15-cqdfn" event={"ID":"aaace599-087f-4a86-91a6-2abd2a65cfaa","Type":"ContainerDied","Data":"0fd57c7c7b533073c70d2f8489cb984fbb3319bfaf92db8e0df6b00310eae21e"} Oct 30 21:49:36.145054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.144975 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fd57c7c7b533073c70d2f8489cb984fbb3319bfaf92db8e0df6b00310eae21e" Oct 30 21:49:36.147688 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.147666 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x" event={"ID":"3ba41b66-8be4-4424-a326-c7e86dc1ffcf","Type":"ContainerStarted","Data":"7bc372061e538c5012f8b75e128820b7563cef283bd86dba23fe159721372049"} Oct 30 21:49:36.182984 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.182951 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8425a8c1-cff8-42cd-af91-c6b3b2f8d222-kube-api-access-f8sqs" pod "8425a8c1-cff8-42cd-af91-c6b3b2f8d222" (UID: "8425a8c1-cff8-42cd-af91-c6b3b2f8d222"). InnerVolumeSpecName "kube-api-access-f8sqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:36.218089 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.218010 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8425a8c1-cff8-42cd-af91-c6b3b2f8d222-maas-api-xks-ca" pod "8425a8c1-cff8-42cd-af91-c6b3b2f8d222" (UID: "8425a8c1-cff8-42cd-af91-c6b3b2f8d222"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:36.229779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.226527 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaace599-087f-4a86-91a6-2abd2a65cfaa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaace599-087f-4a86-91a6-2abd2a65cfaa-maas-api-openshift-ca\") pod \"aaace599-087f-4a86-91a6-2abd2a65cfaa\" (UID: \"aaace599-087f-4a86-91a6-2abd2a65cfaa\") " Oct 30 21:49:36.229779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.226654 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaace599-087f-4a86-91a6-2abd2a65cfaa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaace599-087f-4a86-91a6-2abd2a65cfaa-maas-api-xks-ca\") pod \"aaace599-087f-4a86-91a6-2abd2a65cfaa\" (UID: \"aaace599-087f-4a86-91a6-2abd2a65cfaa\") " Oct 30 21:49:36.237316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.237066 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaace599-087f-4a86-91a6-2abd2a65cfaa-kube-api-access-7j46k\" (UniqueName: \"kubernetes.io/projected/aaace599-087f-4a86-91a6-2abd2a65cfaa-kube-api-access-7j46k\") pod \"aaace599-087f-4a86-91a6-2abd2a65cfaa\" (UID: \"aaace599-087f-4a86-91a6-2abd2a65cfaa\") " Oct 30 21:49:36.244837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.238417 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8425a8c1-cff8-42cd-af91-c6b3b2f8d222-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:36.244837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.238452 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8425a8c1-cff8-42cd-af91-c6b3b2f8d222-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:36.244837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.238468 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8sqs\" (UniqueName: \"kubernetes.io/projected/8425a8c1-cff8-42cd-af91-c6b3b2f8d222-kube-api-access-f8sqs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:36.244837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.239399 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaace599-087f-4a86-91a6-2abd2a65cfaa-maas-api-openshift-ca" pod "aaace599-087f-4a86-91a6-2abd2a65cfaa" (UID: "aaace599-087f-4a86-91a6-2abd2a65cfaa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:36.248125 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.248083 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaace599-087f-4a86-91a6-2abd2a65cfaa-maas-api-xks-ca" pod "aaace599-087f-4a86-91a6-2abd2a65cfaa" (UID: "aaace599-087f-4a86-91a6-2abd2a65cfaa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:36.324796 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.276518 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:36.359080 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.325336 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaace599-087f-4a86-91a6-2abd2a65cfaa-kube-api-access-7j46k" pod "aaace599-087f-4a86-91a6-2abd2a65cfaa" (UID: "aaace599-087f-4a86-91a6-2abd2a65cfaa"). InnerVolumeSpecName "kube-api-access-7j46k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:36.410864 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.361860 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaace599-087f-4a86-91a6-2abd2a65cfaa-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:36.410864 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.361894 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j46k\" (UniqueName: \"kubernetes.io/projected/aaace599-087f-4a86-91a6-2abd2a65cfaa-kube-api-access-7j46k\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:36.410864 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.361914 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaace599-087f-4a86-91a6-2abd2a65cfaa-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:36.526778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.509131 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9"] Oct 30 21:49:36.535935 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.535894 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:49:36.536246 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.535985 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:49:36.536246 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.536025 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:49:36.724954 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:36.724926 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:37.213395 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:37.213359 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:37.224262 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:37.223190 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" event={"ID":"239d3785-9a31-40ed-9e63-2d7e1265b5a2","Type":"ContainerStarted","Data":"6c3d90cae6558521bde9d8ddb74aec810038ed575c9b634e214e3158f58a9763"} Oct 30 21:49:37.224262 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:37.223295 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" event={"ID":"239d3785-9a31-40ed-9e63-2d7e1265b5a2","Type":"ContainerStarted","Data":"307f28457d4c802b56902776fe37616caf91a01ad0521b8e8de9448c326f9db6"} Oct 30 21:49:37.227103 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:37.227069 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x" event={"ID":"3ba41b66-8be4-4424-a326-c7e86dc1ffcf","Type":"ContainerStarted","Data":"e67dcfd46e67df02436914353a77a8d4d8a5c62dc688f7586d8972d374aa23a7"} Oct 30 21:49:37.332614 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:37.332522 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" podStartSLOduration=4.332282044 podStartE2EDuration="4.332282044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:37.271570381 +0000 UTC m=+1245.867387075" watchObservedRunningTime="2026-10-30 21:49:37.332282044 +0000 UTC m=+1245.928098705" Oct 30 21:49:37.348196 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:37.337678 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x" podStartSLOduration=5.337661388 podStartE2EDuration="5.337661388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:37.300412107 +0000 UTC m=+1245.896228772" watchObservedRunningTime="2026-10-30 21:49:37.337661388 +0000 UTC m=+1245.933478048" Oct 30 21:49:38.151694 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:38.151659 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:38.203936 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:38.203859 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:38.216639 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:38.216360 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:38.240963 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:38.240926 5801 generic.go:356] "Generic (PLEG): container finished" podID="3ba41b66-8be4-4424-a326-c7e86dc1ffcf" containerID="e67dcfd46e67df02436914353a77a8d4d8a5c62dc688f7586d8972d374aa23a7" exitCode=0 Oct 30 21:49:38.241157 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:38.241080 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x" event={"ID":"3ba41b66-8be4-4424-a326-c7e86dc1ffcf","Type":"ContainerDied","Data":"e67dcfd46e67df02436914353a77a8d4d8a5c62dc688f7586d8972d374aa23a7"} Oct 30 21:49:38.241543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:38.241520 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:38.242372 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:38.242352 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:49:38.916251 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:38.916212 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:39.285517 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:39.285485 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:39.440812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:39.439288 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:39.484801 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:39.484769 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm"] Oct 30 21:49:39.487669 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:39.487642 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5bf8-jwvbm"] Oct 30 21:49:39.531979 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:39.531941 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e0c8943-d0ef-4c44-8757-e25e5c8b514e" path="/var/lib/kubelet/pods/7e0c8943-d0ef-4c44-8757-e25e5c8b514e/volumes" Oct 30 21:49:39.546705 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:39.546635 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x" Oct 30 21:49:39.549046 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:39.549019 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:39.654745 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:39.654711 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ba41b66-8be4-4424-a326-c7e86dc1ffcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba41b66-8be4-4424-a326-c7e86dc1ffcf-maas-api-xks-ca\") pod \"3ba41b66-8be4-4424-a326-c7e86dc1ffcf\" (UID: \"3ba41b66-8be4-4424-a326-c7e86dc1ffcf\") " Oct 30 21:49:39.654952 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:39.654880 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ba41b66-8be4-4424-a326-c7e86dc1ffcf-kube-api-access-76tzx\" (UniqueName: \"kubernetes.io/projected/3ba41b66-8be4-4424-a326-c7e86dc1ffcf-kube-api-access-76tzx\") pod \"3ba41b66-8be4-4424-a326-c7e86dc1ffcf\" (UID: \"3ba41b66-8be4-4424-a326-c7e86dc1ffcf\") " Oct 30 21:49:39.654952 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:39.654948 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ba41b66-8be4-4424-a326-c7e86dc1ffcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba41b66-8be4-4424-a326-c7e86dc1ffcf-maas-api-openshift-ca\") pod \"3ba41b66-8be4-4424-a326-c7e86dc1ffcf\" (UID: \"3ba41b66-8be4-4424-a326-c7e86dc1ffcf\") " Oct 30 21:49:39.655451 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:39.655421 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba41b66-8be4-4424-a326-c7e86dc1ffcf-maas-api-openshift-ca" pod "3ba41b66-8be4-4424-a326-c7e86dc1ffcf" (UID: "3ba41b66-8be4-4424-a326-c7e86dc1ffcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:39.655683 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:39.655573 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba41b66-8be4-4424-a326-c7e86dc1ffcf-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:39.657414 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:39.657381 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ba41b66-8be4-4424-a326-c7e86dc1ffcf-maas-api-xks-ca" pod "3ba41b66-8be4-4424-a326-c7e86dc1ffcf" (UID: "3ba41b66-8be4-4424-a326-c7e86dc1ffcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:39.657506 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:39.657389 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba41b66-8be4-4424-a326-c7e86dc1ffcf-kube-api-access-76tzx" pod "3ba41b66-8be4-4424-a326-c7e86dc1ffcf" (UID: "3ba41b66-8be4-4424-a326-c7e86dc1ffcf"). InnerVolumeSpecName "kube-api-access-76tzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:39.769819 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:39.757284 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba41b66-8be4-4424-a326-c7e86dc1ffcf-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:39.769819 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:39.757329 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76tzx\" (UniqueName: \"kubernetes.io/projected/3ba41b66-8be4-4424-a326-c7e86dc1ffcf-kube-api-access-76tzx\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:39.943111 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:39.943069 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:40.102312 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.102270 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:40.298547 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.298465 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x" Oct 30 21:49:40.299036 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.298460 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x" event={"ID":"3ba41b66-8be4-4424-a326-c7e86dc1ffcf","Type":"ContainerDied","Data":"7bc372061e538c5012f8b75e128820b7563cef283bd86dba23fe159721372049"} Oct 30 21:49:40.299036 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.298596 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bc372061e538c5012f8b75e128820b7563cef283bd86dba23fe159721372049" Oct 30 21:49:40.345414 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.345374 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9"] Oct 30 21:49:40.346564 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.346533 5801 cpu_manager.go:414] "containerMap: removing container" podUID="aaace599-087f-4a86-91a6-2abd2a65cfaa" containerName="revoke-keys" Oct 30 21:49:40.346564 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.346562 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaace599-087f-4a86-91a6-2abd2a65cfaa" containerName="revoke-keys" Oct 30 21:49:40.346928 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.346606 5801 cpu_manager.go:414] "containerMap: removing container" podUID="8425a8c1-cff8-42cd-af91-c6b3b2f8d222" containerName="revoke-keys" Oct 30 21:49:40.346928 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.346616 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8425a8c1-cff8-42cd-af91-c6b3b2f8d222" containerName="revoke-keys" Oct 30 21:49:40.346928 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.346626 5801 cpu_manager.go:414] "containerMap: removing container" podUID="3ba41b66-8be4-4424-a326-c7e86dc1ffcf" containerName="revoke-keys" Oct 30 21:49:40.346928 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.346633 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ba41b66-8be4-4424-a326-c7e86dc1ffcf" containerName="revoke-keys" Oct 30 21:49:40.346928 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.346854 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaace599-087f-4a86-91a6-2abd2a65cfaa" containerName="revoke-keys" Oct 30 21:49:40.346928 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.346874 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="8425a8c1-cff8-42cd-af91-c6b3b2f8d222" containerName="revoke-keys" Oct 30 21:49:40.346928 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.346882 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ba41b66-8be4-4424-a326-c7e86dc1ffcf" containerName="revoke-keys" Oct 30 21:49:40.350399 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.350284 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:40.351038 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.351015 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.394770 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.383493 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9"] Oct 30 21:49:40.469263 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.469228 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:40.476997 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.476971 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.477117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.477015 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.477117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.477040 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.477194 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.477120 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.477262 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.477248 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.477401 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.477386 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fcvm\" (UniqueName: \"kubernetes.io/projected/4184fd8e-d9fb-474d-95b8-95afb19dac0b-kube-api-access-4fcvm\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.477468 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.477431 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.477509 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.477472 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.477591 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.477567 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.579556 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.579512 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.579711 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.579695 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fcvm\" (UniqueName: \"kubernetes.io/projected/4184fd8e-d9fb-474d-95b8-95afb19dac0b-kube-api-access-4fcvm\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.579806 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.579746 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.579864 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.579827 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.579917 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.579876 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.579992 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.579969 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-workload-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.580064 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.579989 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.580117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.580058 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.580117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.580090 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.580227 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.580136 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.580849 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.580552 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-data\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.580849 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.580746 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istiod-ca-cert\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.580849 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.580795 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-workload-certs\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.581109 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.580882 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-credential-socket\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.582275 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.582255 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-envoy\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.582825 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.582808 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-podinfo\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.594200 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.594171 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-token\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.602892 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.602872 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fcvm\" (UniqueName: \"kubernetes.io/projected/4184fd8e-d9fb-474d-95b8-95afb19dac0b-kube-api-access-4fcvm\") pod \"e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.686471 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.683839 5801 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" secret="" err="secret \"e2e-worker-w6-7dfff0-openshift-default-dockercfg-lrx4l\" not found" Oct 30 21:49:40.686471 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.683974 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:40.694200 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.691480 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps"] Oct 30 21:49:40.694200 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.691680 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" podUID="8aa256c9-7e49-4144-9c43-91c8cdca8a7a" containerName="istio-proxy" containerID="cri-o://9386397b7b5ab358b2edd8b811251d9f1c14af3554fbebde08ccf1b9b75fce15" gracePeriod=30 Oct 30 21:49:40.701299 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.701265 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9"] Oct 30 21:49:40.714872 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.714842 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k"] Oct 30 21:49:40.715120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.715095 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" podUID="528a2150-a9c8-42d4-8203-113bc618287a" containerName="istio-proxy" containerID="cri-o://3eec4a313b1e11242469d9a71c03c5223525b005268d8da576e618346936b9c7" gracePeriod=30 Oct 30 21:49:40.725921 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.723647 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk"] Oct 30 21:49:40.725921 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.723898 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" podUID="b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" containerName="istio-proxy" containerID="cri-o://fc31a7714628ad17f8e7d9b3a1d9f880d34295fce5cb38456fa06155c669ff1b" gracePeriod=30 Oct 30 21:49:40.742838 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:40.742785 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:41.003576 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.003427 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:41.048764 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.048717 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9"] Oct 30 21:49:41.079302 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:49:41.079263 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4184fd8e_d9fb_474d_95b8_95afb19dac0b.slice/crio-f0c3459116a6fcb79b4a177486dd83b2799524027e7d21b7d28ae76500be5861 WatchSource:0}: Error finding container f0c3459116a6fcb79b4a177486dd83b2799524027e7d21b7d28ae76500be5861: Status 404 returned error can't find the container with id f0c3459116a6fcb79b4a177486dd83b2799524027e7d21b7d28ae76500be5861 Oct 30 21:49:41.085344 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.084171 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:49:41.085344 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.084271 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:49:41.085344 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.084319 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:49:41.330719 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.330670 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" event={"ID":"4184fd8e-d9fb-474d-95b8-95afb19dac0b","Type":"ContainerStarted","Data":"ad1c82ca03e906c32be1c4c0c5ef5f134cde5d73c9022e761ae7dedd5038f81b"} Oct 30 21:49:41.331352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.330820 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" event={"ID":"4184fd8e-d9fb-474d-95b8-95afb19dac0b","Type":"ContainerStarted","Data":"f0c3459116a6fcb79b4a177486dd83b2799524027e7d21b7d28ae76500be5861"} Oct 30 21:49:41.331352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.331010 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" podUID="4184fd8e-d9fb-474d-95b8-95afb19dac0b" containerName="istio-proxy" containerID="cri-o://ad1c82ca03e906c32be1c4c0c5ef5f134cde5d73c9022e761ae7dedd5038f81b" gracePeriod=30 Oct 30 21:49:41.334075 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.334038 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:41.383201 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.383157 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84"] Oct 30 21:49:41.424009 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.423975 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84"] Oct 30 21:49:41.424312 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.424295 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84" Oct 30 21:49:41.432377 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.432351 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f608f7-dockercfg-4r5lv\"" Oct 30 21:49:41.435208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.435171 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f608f7\"" Oct 30 21:49:41.443557 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.436914 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" podStartSLOduration=1.436896947 podStartE2EDuration="1.436896947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:41.370997741 +0000 UTC m=+1249.966814402" watchObservedRunningTime="2026-10-30 21:49:41.436896947 +0000 UTC m=+1250.032713609" Oct 30 21:49:41.461369 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.457279 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:41.468313 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.467743 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9"] Oct 30 21:49:41.489963 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.489924 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9"] Oct 30 21:49:41.490329 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.490311 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9" Oct 30 21:49:41.507725 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.507689 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a88411-dade-4b2f-98a1-24675c120cad-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9\" (UID: \"99a88411-dade-4b2f-98a1-24675c120cad\") " pod="openshift-ingress/payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9" Oct 30 21:49:41.507909 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.507793 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgcvp\" (UniqueName: \"kubernetes.io/projected/e95bb984-7abc-49e6-acb7-5250d1b4d98c-kube-api-access-pgcvp\") pod \"payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84\" (UID: \"e95bb984-7abc-49e6-acb7-5250d1b4d98c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84" Oct 30 21:49:41.507909 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.507863 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdz7l\" (UniqueName: \"kubernetes.io/projected/99a88411-dade-4b2f-98a1-24675c120cad-kube-api-access-jdz7l\") pod \"payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9\" (UID: \"99a88411-dade-4b2f-98a1-24675c120cad\") " pod="openshift-ingress/payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9" Oct 30 21:49:41.508038 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.507922 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95bb984-7abc-49e6-acb7-5250d1b4d98c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84\" (UID: \"e95bb984-7abc-49e6-acb7-5250d1b4d98c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84" Oct 30 21:49:41.616843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.613647 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a88411-dade-4b2f-98a1-24675c120cad-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9\" (UID: \"99a88411-dade-4b2f-98a1-24675c120cad\") " pod="openshift-ingress/payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9" Oct 30 21:49:41.616843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.613774 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgcvp\" (UniqueName: \"kubernetes.io/projected/e95bb984-7abc-49e6-acb7-5250d1b4d98c-kube-api-access-pgcvp\") pod \"payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84\" (UID: \"e95bb984-7abc-49e6-acb7-5250d1b4d98c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84" Oct 30 21:49:41.616843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.613861 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdz7l\" (UniqueName: \"kubernetes.io/projected/99a88411-dade-4b2f-98a1-24675c120cad-kube-api-access-jdz7l\") pod \"payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9\" (UID: \"99a88411-dade-4b2f-98a1-24675c120cad\") " pod="openshift-ingress/payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9" Oct 30 21:49:41.616843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.613931 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95bb984-7abc-49e6-acb7-5250d1b4d98c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84\" (UID: \"e95bb984-7abc-49e6-acb7-5250d1b4d98c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84" Oct 30 21:49:41.616843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.615215 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95bb984-7abc-49e6-acb7-5250d1b4d98c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84\" (UID: \"e95bb984-7abc-49e6-acb7-5250d1b4d98c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84" Oct 30 21:49:41.616843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.615965 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a88411-dade-4b2f-98a1-24675c120cad-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9\" (UID: \"99a88411-dade-4b2f-98a1-24675c120cad\") " pod="openshift-ingress/payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9" Oct 30 21:49:41.657122 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.657033 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdz7l\" (UniqueName: \"kubernetes.io/projected/99a88411-dade-4b2f-98a1-24675c120cad-kube-api-access-jdz7l\") pod \"payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9\" (UID: \"99a88411-dade-4b2f-98a1-24675c120cad\") " pod="openshift-ingress/payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9" Oct 30 21:49:41.659731 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.659692 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgcvp\" (UniqueName: \"kubernetes.io/projected/e95bb984-7abc-49e6-acb7-5250d1b4d98c-kube-api-access-pgcvp\") pod \"payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84\" (UID: \"e95bb984-7abc-49e6-acb7-5250d1b4d98c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84" Oct 30 21:49:41.783150 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.783099 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84" Oct 30 21:49:41.823803 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.822703 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9" Oct 30 21:49:41.945310 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:41.944996 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:42.137074 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:42.136985 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84"] Oct 30 21:49:42.149152 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:42.149124 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:49:42.149300 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:42.149172 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" podUID="528a2150-a9c8-42d4-8203-113bc618287a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:49:42.201413 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:42.201239 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9"] Oct 30 21:49:42.347465 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:42.347425 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84" event={"ID":"e95bb984-7abc-49e6-acb7-5250d1b4d98c","Type":"ContainerStarted","Data":"4446252aa2c572e360073011dc8d4218313f52c9f0b26083c86ab92023c86d71"} Oct 30 21:49:42.348015 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:42.347491 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84" event={"ID":"e95bb984-7abc-49e6-acb7-5250d1b4d98c","Type":"ContainerStarted","Data":"6281ac2c83001faba83daadc590d8d49bc63d93ba377e1f33f282075ed23f8c1"} Oct 30 21:49:42.348567 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:42.348544 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84" Oct 30 21:49:42.351710 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:42.351675 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9" event={"ID":"99a88411-dade-4b2f-98a1-24675c120cad","Type":"ContainerStarted","Data":"da21626418113e945967fcf70fbf4accbeba0123b684af4440575a4508f26d78"} Oct 30 21:49:42.351872 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:42.351796 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9" event={"ID":"99a88411-dade-4b2f-98a1-24675c120cad","Type":"ContainerStarted","Data":"164c55e5b89417c17b9486f518880eda9b7adf9ecc0d2f70d65f6173fc3e5555"} Oct 30 21:49:42.352547 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:42.352528 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9" Oct 30 21:49:42.370408 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:42.370361 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84" podStartSLOduration=1.37034898 podStartE2EDuration="1.37034898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:42.367943471 +0000 UTC m=+1250.963760135" watchObservedRunningTime="2026-10-30 21:49:42.37034898 +0000 UTC m=+1250.966165630" Oct 30 21:49:42.393821 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:42.393765 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9" podStartSLOduration=1.3937200459999999 podStartE2EDuration="1.393720046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:42.388931304 +0000 UTC m=+1250.984747984" watchObservedRunningTime="2026-10-30 21:49:42.393720046 +0000 UTC m=+1250.989536707" Oct 30 21:49:42.855075 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:42.855036 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:43.158152 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:43.158077 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4"] Oct 30 21:49:43.175323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:43.175197 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7"] Oct 30 21:49:43.254839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:43.254747 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:43.369098 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:43.369064 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:43.813988 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:43.813956 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt"] Oct 30 21:49:43.820393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:43.820366 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:43.843358 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:43.843319 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt"] Oct 30 21:49:43.849707 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:43.848507 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:43.981883 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:43.981794 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-workload-certs\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:43.982065 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:43.981952 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-istio-envoy\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:43.982065 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:43.981999 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409f33be-0503-4f36-ad79-93d1646f51ec-istio-podinfo\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:43.982203 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:43.982145 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t25v\" (UniqueName: \"kubernetes.io/projected/409f33be-0503-4f36-ad79-93d1646f51ec-kube-api-access-7t25v\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:43.982349 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:43.982324 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-workload-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:43.982408 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:43.982373 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-credential-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:43.982469 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:43.982421 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409f33be-0503-4f36-ad79-93d1646f51ec-istiod-ca-cert\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:43.982553 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:43.982502 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-istio-data\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:43.982614 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:43.982598 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409f33be-0503-4f36-ad79-93d1646f51ec-istio-token\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:44.084369 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.084333 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409f33be-0503-4f36-ad79-93d1646f51ec-istio-token\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:44.084580 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.084397 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-workload-certs\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:44.084668 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.084598 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-istio-envoy\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:44.084740 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.084697 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409f33be-0503-4f36-ad79-93d1646f51ec-istio-podinfo\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:44.084822 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.084806 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t25v\" (UniqueName: \"kubernetes.io/projected/409f33be-0503-4f36-ad79-93d1646f51ec-kube-api-access-7t25v\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:44.084909 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.084886 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-workload-certs\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:44.085018 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.084997 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-workload-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:44.085096 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.085043 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-credential-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:44.085096 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.085074 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409f33be-0503-4f36-ad79-93d1646f51ec-istiod-ca-cert\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:44.085211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.085129 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-istio-data\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:44.085601 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.085531 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-workload-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:44.085714 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.085599 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-istio-data\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:44.085835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.085807 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409f33be-0503-4f36-ad79-93d1646f51ec-istiod-ca-cert\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:44.086092 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.086070 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-credential-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:44.087816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.087775 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-istio-envoy\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:44.088141 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.088122 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409f33be-0503-4f36-ad79-93d1646f51ec-istio-podinfo\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:44.097316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.097295 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409f33be-0503-4f36-ad79-93d1646f51ec-istio-token\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:44.097402 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.097343 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t25v\" (UniqueName: \"kubernetes.io/projected/409f33be-0503-4f36-ad79-93d1646f51ec-kube-api-access-7t25v\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:44.159329 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.159296 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:44.182633 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.182603 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:44.224804 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.224725 5801 patch_prober.go:29] interesting pod/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 21:49:44.224978 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.224857 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" podUID="f9975b4c-127e-4130-ae64-b8785b0acc7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 21:49:44.373306 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.373274 5801 generic.go:356] "Generic (PLEG): container finished" podID="f9975b4c-127e-4130-ae64-b8785b0acc7f" containerID="cb9ad72496a0cd9cbd1dd57911db06b323ed7484e3511285d9020bed8a079349" exitCode=0 Oct 30 21:49:44.373803 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.373348 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" event={"ID":"f9975b4c-127e-4130-ae64-b8785b0acc7f","Type":"ContainerDied","Data":"cb9ad72496a0cd9cbd1dd57911db06b323ed7484e3511285d9020bed8a079349"} Oct 30 21:49:44.373962 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.373946 5801 scope.go:122] "RemoveContainer" containerID="cb9ad72496a0cd9cbd1dd57911db06b323ed7484e3511285d9020bed8a079349" Oct 30 21:49:44.375721 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.375691 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:49:44.375858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.375789 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:49:44.375858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.375824 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:49:44.446644 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.445797 5801 patch_prober.go:29] interesting pod/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:49:44.446644 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.445895 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" podUID="8aa256c9-7e49-4144-9c43-91c8cdca8a7a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:49:44.456506 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:49:44.456460 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409f33be_0503_4f36_ad79_93d1646f51ec.slice/crio-74bce5f3595a76c9f5816df453ebc733526d1764336d4279a442f7dbe64114e4 WatchSource:0}: Error finding container 74bce5f3595a76c9f5816df453ebc733526d1764336d4279a442f7dbe64114e4: Status 404 returned error can't find the container with id 74bce5f3595a76c9f5816df453ebc733526d1764336d4279a442f7dbe64114e4 Oct 30 21:49:44.459867 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.459447 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:49:44.459867 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.459513 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:49:44.459867 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.459541 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:49:44.462205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.462182 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt"] Oct 30 21:49:44.773469 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:44.773378 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:45.548722 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:45.548688 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" event={"ID":"f9975b4c-127e-4130-ae64-b8785b0acc7f","Type":"ContainerStarted","Data":"6413883aa622617e7e9324d67928ea69d7313b63d4be87ca5628bf65627aabc8"} Oct 30 21:49:45.549308 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:45.548737 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" event={"ID":"409f33be-0503-4f36-ad79-93d1646f51ec","Type":"ContainerStarted","Data":"0b29388fe91c2329a24c37d18faade709132c7a3946366de57909f4659b79126"} Oct 30 21:49:45.549308 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:45.548772 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" event={"ID":"409f33be-0503-4f36-ad79-93d1646f51ec","Type":"ContainerStarted","Data":"74bce5f3595a76c9f5816df453ebc733526d1764336d4279a442f7dbe64114e4"} Oct 30 21:49:45.603369 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:45.603297 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" podStartSLOduration=2.60327816 podStartE2EDuration="2.60327816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:45.587089959 +0000 UTC m=+1254.182906644" watchObservedRunningTime="2026-10-30 21:49:45.60327816 +0000 UTC m=+1254.199094825" Oct 30 21:49:45.836914 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:45.836882 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:45.911195 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:45.898770 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:49:45.911195 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:45.898894 5801 patch_prober.go:29] interesting pod/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 21:49:45.911195 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:45.898944 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" podUID="f9975b4c-127e-4130-ae64-b8785b0acc7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 21:49:46.212833 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.191841 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:46.220629 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.217892 5801 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 21:49:46.220629 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.217988 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" podUID="409f33be-0503-4f36-ad79-93d1646f51ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 21:49:46.537368 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.536996 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:49:46.543936 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.543910 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:49:46.550624 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.550600 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:49:46.621227 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.621186 5801 generic.go:356] "Generic (PLEG): container finished" podID="8aa256c9-7e49-4144-9c43-91c8cdca8a7a" containerID="9386397b7b5ab358b2edd8b811251d9f1c14af3554fbebde08ccf1b9b75fce15" exitCode=0 Oct 30 21:49:46.621398 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.621316 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" Oct 30 21:49:46.621398 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.621346 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" event={"ID":"8aa256c9-7e49-4144-9c43-91c8cdca8a7a","Type":"ContainerDied","Data":"9386397b7b5ab358b2edd8b811251d9f1c14af3554fbebde08ccf1b9b75fce15"} Oct 30 21:49:46.621398 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.621395 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps" event={"ID":"8aa256c9-7e49-4144-9c43-91c8cdca8a7a","Type":"ContainerDied","Data":"20b3f53a9fbe0f0fd7acc1ea47040e4f48bcce2df8905d05e16b7b9d35da496e"} Oct 30 21:49:46.621593 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.621423 5801 scope.go:122] "RemoveContainer" containerID="9386397b7b5ab358b2edd8b811251d9f1c14af3554fbebde08ccf1b9b75fce15" Oct 30 21:49:46.624171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.624142 5801 generic.go:356] "Generic (PLEG): container finished" podID="4184fd8e-d9fb-474d-95b8-95afb19dac0b" containerID="ad1c82ca03e906c32be1c4c0c5ef5f134cde5d73c9022e761ae7dedd5038f81b" exitCode=0 Oct 30 21:49:46.624294 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.624228 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" event={"ID":"4184fd8e-d9fb-474d-95b8-95afb19dac0b","Type":"ContainerDied","Data":"ad1c82ca03e906c32be1c4c0c5ef5f134cde5d73c9022e761ae7dedd5038f81b"} Oct 30 21:49:46.630915 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.630884 5801 generic.go:356] "Generic (PLEG): container finished" podID="b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" containerID="fc31a7714628ad17f8e7d9b3a1d9f880d34295fce5cb38456fa06155c669ff1b" exitCode=0 Oct 30 21:49:46.631077 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.631055 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" event={"ID":"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7","Type":"ContainerDied","Data":"fc31a7714628ad17f8e7d9b3a1d9f880d34295fce5cb38456fa06155c669ff1b"} Oct 30 21:49:46.631156 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.631109 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" event={"ID":"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7","Type":"ContainerDied","Data":"17417a7b858207a41b0cd6690be4fbf42099c10b30c0febd84d31c6e7cab9984"} Oct 30 21:49:46.631220 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.631198 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk" Oct 30 21:49:46.633503 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.633473 5801 generic.go:356] "Generic (PLEG): container finished" podID="528a2150-a9c8-42d4-8203-113bc618287a" containerID="3eec4a313b1e11242469d9a71c03c5223525b005268d8da576e618346936b9c7" exitCode=0 Oct 30 21:49:46.633613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.633511 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" Oct 30 21:49:46.633675 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.633634 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" event={"ID":"528a2150-a9c8-42d4-8203-113bc618287a","Type":"ContainerDied","Data":"3eec4a313b1e11242469d9a71c03c5223525b005268d8da576e618346936b9c7"} Oct 30 21:49:46.633675 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.633661 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k" event={"ID":"528a2150-a9c8-42d4-8203-113bc618287a","Type":"ContainerDied","Data":"bf230e2376045525159c8b7653515ab77629c56d2c572077c54491af9ec2b087"} Oct 30 21:49:46.647718 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.641504 5801 scope.go:122] "RemoveContainer" containerID="5eff4855aa7a6356c524849160e4af14d893fcf75713e393ff74f13aaffd3211" Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.648055 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-kube-api-access-nkxck\" (UniqueName: \"kubernetes.io/projected/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-kube-api-access-nkxck\") pod \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.648136 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-workload-certs\") pod \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.648203 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-workload-certs\") pod \"528a2150-a9c8-42d4-8203-113bc618287a\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.648249 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-istio-envoy\") pod \"528a2150-a9c8-42d4-8203-113bc618287a\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.648288 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istiod-ca-cert\") pod \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.648322 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-workload-socket\") pod \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.648359 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528a2150-a9c8-42d4-8203-113bc618287a-kube-api-access-6dr6r\" (UniqueName: \"kubernetes.io/projected/528a2150-a9c8-42d4-8203-113bc618287a-kube-api-access-6dr6r\") pod \"528a2150-a9c8-42d4-8203-113bc618287a\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.648406 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-workload-socket\") pod \"528a2150-a9c8-42d4-8203-113bc618287a\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.648444 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-podinfo\") pod \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.648491 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-token\" (UniqueName: \"kubernetes.io/projected/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-token\") pod \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.648531 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528a2150-a9c8-42d4-8203-113bc618287a-istio-token\" (UniqueName: \"kubernetes.io/projected/528a2150-a9c8-42d4-8203-113bc618287a-istio-token\") pod \"528a2150-a9c8-42d4-8203-113bc618287a\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.648626 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-credential-socket\") pod \"528a2150-a9c8-42d4-8203-113bc618287a\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.648658 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-envoy\") pod \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.648692 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-credential-socket\") pod \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.648738 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-workload-socket\") pod \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.648784 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-podinfo\") pod \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.648943 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/528a2150-a9c8-42d4-8203-113bc618287a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528a2150-a9c8-42d4-8203-113bc618287a-istiod-ca-cert\") pod \"528a2150-a9c8-42d4-8203-113bc618287a\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.649001 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-data\") pod \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.649031 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istiod-ca-cert\") pod \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.649072 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-envoy\") pod \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.649169 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-workload-certs\") pod \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.649197 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/528a2150-a9c8-42d4-8203-113bc618287a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528a2150-a9c8-42d4-8203-113bc618287a-istio-podinfo\") pod \"528a2150-a9c8-42d4-8203-113bc618287a\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.649249 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-token\" (UniqueName: \"kubernetes.io/projected/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-token\") pod \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.649305 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-istio-data\") pod \"528a2150-a9c8-42d4-8203-113bc618287a\" (UID: \"528a2150-a9c8-42d4-8203-113bc618287a\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.649380 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-credential-socket\") pod \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.649421 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-data\") pod \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " Oct 30 21:49:46.649582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.649494 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-kube-api-access-56r4q\" (UniqueName: \"kubernetes.io/projected/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-kube-api-access-56r4q\") pod \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " Oct 30 21:49:46.668326 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.651743 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-workload-certs" pod "528a2150-a9c8-42d4-8203-113bc618287a" (UID: "528a2150-a9c8-42d4-8203-113bc618287a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:46.668326 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.656833 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-workload-socket" pod "8aa256c9-7e49-4144-9c43-91c8cdca8a7a" (UID: "8aa256c9-7e49-4144-9c43-91c8cdca8a7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:46.668326 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.662982 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istiod-ca-cert" pod "8aa256c9-7e49-4144-9c43-91c8cdca8a7a" (UID: "8aa256c9-7e49-4144-9c43-91c8cdca8a7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:46.668326 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.663265 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-data" pod "8aa256c9-7e49-4144-9c43-91c8cdca8a7a" (UID: "8aa256c9-7e49-4144-9c43-91c8cdca8a7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:46.668326 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.663597 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528a2150-a9c8-42d4-8203-113bc618287a-istiod-ca-cert" pod "528a2150-a9c8-42d4-8203-113bc618287a" (UID: "528a2150-a9c8-42d4-8203-113bc618287a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:46.718469 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.718102 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-workload-certs" pod "8aa256c9-7e49-4144-9c43-91c8cdca8a7a" (UID: "8aa256c9-7e49-4144-9c43-91c8cdca8a7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:46.722346 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.721168 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-credential-socket" pod "b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" (UID: "b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:46.722346 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.721355 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-istio-data" pod "528a2150-a9c8-42d4-8203-113bc618287a" (UID: "528a2150-a9c8-42d4-8203-113bc618287a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:46.723063 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.723022 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-data" pod "b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" (UID: "b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:46.723209 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.723178 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-workload-socket" pod "528a2150-a9c8-42d4-8203-113bc618287a" (UID: "528a2150-a9c8-42d4-8203-113bc618287a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:46.728880 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.728681 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-credential-socket" pod "8aa256c9-7e49-4144-9c43-91c8cdca8a7a" (UID: "8aa256c9-7e49-4144-9c43-91c8cdca8a7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:46.729139 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.729042 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-workload-socket" pod "b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" (UID: "b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:46.729139 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.729116 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/528a2150-a9c8-42d4-8203-113bc618287a-istio-podinfo" pod "528a2150-a9c8-42d4-8203-113bc618287a" (UID: "528a2150-a9c8-42d4-8203-113bc618287a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:49:46.730744 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.730273 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-credential-socket" pod "528a2150-a9c8-42d4-8203-113bc618287a" (UID: "528a2150-a9c8-42d4-8203-113bc618287a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:46.730744 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.730451 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istiod-ca-cert" pod "b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" (UID: "b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:46.731922 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.731831 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-workload-certs" pod "b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" (UID: "b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:46.735926 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.735904 5801 scope.go:122] "RemoveContainer" containerID="9386397b7b5ab358b2edd8b811251d9f1c14af3554fbebde08ccf1b9b75fce15" Oct 30 21:49:46.736649 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:49:46.736529 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9386397b7b5ab358b2edd8b811251d9f1c14af3554fbebde08ccf1b9b75fce15\": container with ID starting with 9386397b7b5ab358b2edd8b811251d9f1c14af3554fbebde08ccf1b9b75fce15 not found: ID does not exist" containerID="9386397b7b5ab358b2edd8b811251d9f1c14af3554fbebde08ccf1b9b75fce15" Oct 30 21:49:46.736649 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.736573 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9386397b7b5ab358b2edd8b811251d9f1c14af3554fbebde08ccf1b9b75fce15"} err="failed to get container status \"9386397b7b5ab358b2edd8b811251d9f1c14af3554fbebde08ccf1b9b75fce15\": rpc error: code = NotFound desc = could not find container \"9386397b7b5ab358b2edd8b811251d9f1c14af3554fbebde08ccf1b9b75fce15\": container with ID starting with 9386397b7b5ab358b2edd8b811251d9f1c14af3554fbebde08ccf1b9b75fce15 not found: ID does not exist" Oct 30 21:49:46.736649 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.736601 5801 scope.go:122] "RemoveContainer" containerID="5eff4855aa7a6356c524849160e4af14d893fcf75713e393ff74f13aaffd3211" Oct 30 21:49:46.737269 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:49:46.737047 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eff4855aa7a6356c524849160e4af14d893fcf75713e393ff74f13aaffd3211\": container with ID starting with 5eff4855aa7a6356c524849160e4af14d893fcf75713e393ff74f13aaffd3211 not found: ID does not exist" containerID="5eff4855aa7a6356c524849160e4af14d893fcf75713e393ff74f13aaffd3211" Oct 30 21:49:46.737269 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.737085 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eff4855aa7a6356c524849160e4af14d893fcf75713e393ff74f13aaffd3211"} err="failed to get container status \"5eff4855aa7a6356c524849160e4af14d893fcf75713e393ff74f13aaffd3211\": rpc error: code = NotFound desc = could not find container \"5eff4855aa7a6356c524849160e4af14d893fcf75713e393ff74f13aaffd3211\": container with ID starting with 5eff4855aa7a6356c524849160e4af14d893fcf75713e393ff74f13aaffd3211 not found: ID does not exist" Oct 30 21:49:46.737269 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.737108 5801 scope.go:122] "RemoveContainer" containerID="fc31a7714628ad17f8e7d9b3a1d9f880d34295fce5cb38456fa06155c669ff1b" Oct 30 21:49:46.745797 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.744335 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528a2150-a9c8-42d4-8203-113bc618287a-istio-token" pod "528a2150-a9c8-42d4-8203-113bc618287a" (UID: "528a2150-a9c8-42d4-8203-113bc618287a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.751344 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528a2150-a9c8-42d4-8203-113bc618287a-kube-api-access-6dr6r" pod "528a2150-a9c8-42d4-8203-113bc618287a" (UID: "528a2150-a9c8-42d4-8203-113bc618287a"). InnerVolumeSpecName "kube-api-access-6dr6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.751460 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-token" pod "8aa256c9-7e49-4144-9c43-91c8cdca8a7a" (UID: "8aa256c9-7e49-4144-9c43-91c8cdca8a7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.751564 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-envoy" pod "8aa256c9-7e49-4144-9c43-91c8cdca8a7a" (UID: "8aa256c9-7e49-4144-9c43-91c8cdca8a7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.751599 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-podinfo" pod "8aa256c9-7e49-4144-9c43-91c8cdca8a7a" (UID: "8aa256c9-7e49-4144-9c43-91c8cdca8a7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.751691 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-kube-api-access-nkxck" pod "b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" (UID: "b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7"). InnerVolumeSpecName "kube-api-access-nkxck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.751816 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-podinfo\") pod \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.751983 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-envoy\") pod \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.752080 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-token\" (UniqueName: \"kubernetes.io/projected/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-token\") pod \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\" (UID: \"8aa256c9-7e49-4144-9c43-91c8cdca8a7a\") " Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.752195 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-kube-api-access-nkxck\" (UniqueName: \"kubernetes.io/projected/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-kube-api-access-nkxck\") pod \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\" (UID: \"b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7\") " Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.760235 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.760264 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.760281 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dr6r\" (UniqueName: \"kubernetes.io/projected/528a2150-a9c8-42d4-8203-113bc618287a-kube-api-access-6dr6r\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.760296 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.760311 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/528a2150-a9c8-42d4-8203-113bc618287a-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.760325 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.760340 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.760354 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.760369 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528a2150-a9c8-42d4-8203-113bc618287a-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.760383 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.760398 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.760415 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.760430 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528a2150-a9c8-42d4-8203-113bc618287a-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.760444 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.760459 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.760474 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.760489 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.760503 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:49:46.760837 5801 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/8aa256c9-7e49-4144-9c43-91c8cdca8a7a/volumes/kubernetes.io~downward-api/istio-podinfo Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.760854 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-podinfo" pod "8aa256c9-7e49-4144-9c43-91c8cdca8a7a" (UID: "8aa256c9-7e49-4144-9c43-91c8cdca8a7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:49:46.760899 5801 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/8aa256c9-7e49-4144-9c43-91c8cdca8a7a/volumes/kubernetes.io~empty-dir/istio-envoy Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.760909 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-envoy" pod "8aa256c9-7e49-4144-9c43-91c8cdca8a7a" (UID: "8aa256c9-7e49-4144-9c43-91c8cdca8a7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:49:46.760954 5801 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/8aa256c9-7e49-4144-9c43-91c8cdca8a7a/volumes/kubernetes.io~projected/istio-token Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.760996 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-token" pod "8aa256c9-7e49-4144-9c43-91c8cdca8a7a" (UID: "8aa256c9-7e49-4144-9c43-91c8cdca8a7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:49:46.761060 5801 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7/volumes/kubernetes.io~projected/kube-api-access-nkxck Oct 30 21:49:46.785391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.761074 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-kube-api-access-nkxck" pod "b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" (UID: "b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7"). InnerVolumeSpecName "kube-api-access-nkxck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:46.841612 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.819445 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:46.841612 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.819712 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-token" pod "b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" (UID: "b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:46.909254 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.877273 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.909254 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.877312 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.909254 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.877329 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.909254 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.877344 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.909254 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.877359 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkxck\" (UniqueName: \"kubernetes.io/projected/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-kube-api-access-nkxck\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:46.909254 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.899140 5801 patch_prober.go:29] interesting pod/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 21:49:46.909254 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.899211 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" podUID="f9975b4c-127e-4130-ae64-b8785b0acc7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 21:49:46.948272 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.944123 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-kube-api-access-56r4q" pod "8aa256c9-7e49-4144-9c43-91c8cdca8a7a" (UID: "8aa256c9-7e49-4144-9c43-91c8cdca8a7a"). InnerVolumeSpecName "kube-api-access-56r4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:46.948272 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.944261 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-istio-envoy" pod "528a2150-a9c8-42d4-8203-113bc618287a" (UID: "528a2150-a9c8-42d4-8203-113bc618287a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:46.948272 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.944398 5801 scope.go:122] "RemoveContainer" containerID="fc31a7714628ad17f8e7d9b3a1d9f880d34295fce5cb38456fa06155c669ff1b" Oct 30 21:49:46.948272 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.946723 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-envoy" pod "b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" (UID: "b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:46.948272 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.946825 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-podinfo" pod "b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" (UID: "b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:49:46.948272 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:49:46.946917 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc31a7714628ad17f8e7d9b3a1d9f880d34295fce5cb38456fa06155c669ff1b\": container with ID starting with fc31a7714628ad17f8e7d9b3a1d9f880d34295fce5cb38456fa06155c669ff1b not found: ID does not exist" containerID="fc31a7714628ad17f8e7d9b3a1d9f880d34295fce5cb38456fa06155c669ff1b" Oct 30 21:49:46.948272 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.946950 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc31a7714628ad17f8e7d9b3a1d9f880d34295fce5cb38456fa06155c669ff1b"} err="failed to get container status \"fc31a7714628ad17f8e7d9b3a1d9f880d34295fce5cb38456fa06155c669ff1b\": rpc error: code = NotFound desc = could not find container \"fc31a7714628ad17f8e7d9b3a1d9f880d34295fce5cb38456fa06155c669ff1b\": container with ID starting with fc31a7714628ad17f8e7d9b3a1d9f880d34295fce5cb38456fa06155c669ff1b not found: ID does not exist" Oct 30 21:49:46.948272 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.946978 5801 scope.go:122] "RemoveContainer" containerID="3eec4a313b1e11242469d9a71c03c5223525b005268d8da576e618346936b9c7" Oct 30 21:49:47.013837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.996067 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528a2150-a9c8-42d4-8203-113bc618287a-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:47.013837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.996110 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:47.013837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.996126 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:47.013837 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:46.996141 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56r4q\" (UniqueName: \"kubernetes.io/projected/8aa256c9-7e49-4144-9c43-91c8cdca8a7a-kube-api-access-56r4q\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:47.029248 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.029154 5801 scope.go:122] "RemoveContainer" containerID="3eec4a313b1e11242469d9a71c03c5223525b005268d8da576e618346936b9c7" Oct 30 21:49:47.029449 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:49:47.029426 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eec4a313b1e11242469d9a71c03c5223525b005268d8da576e618346936b9c7\": container with ID starting with 3eec4a313b1e11242469d9a71c03c5223525b005268d8da576e618346936b9c7 not found: ID does not exist" containerID="3eec4a313b1e11242469d9a71c03c5223525b005268d8da576e618346936b9c7" Oct 30 21:49:47.029539 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.029456 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eec4a313b1e11242469d9a71c03c5223525b005268d8da576e618346936b9c7"} err="failed to get container status \"3eec4a313b1e11242469d9a71c03c5223525b005268d8da576e618346936b9c7\": rpc error: code = NotFound desc = could not find container \"3eec4a313b1e11242469d9a71c03c5223525b005268d8da576e618346936b9c7\": container with ID starting with 3eec4a313b1e11242469d9a71c03c5223525b005268d8da576e618346936b9c7 not found: ID does not exist" Oct 30 21:49:47.032624 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.032603 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:47.051324 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.051298 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf"] Oct 30 21:49:47.052019 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.052004 5801 cpu_manager.go:414] "containerMap: removing container" podUID="8aa256c9-7e49-4144-9c43-91c8cdca8a7a" containerName="istio-proxy" Oct 30 21:49:47.052063 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.052020 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aa256c9-7e49-4144-9c43-91c8cdca8a7a" containerName="istio-proxy" Oct 30 21:49:47.052063 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.052039 5801 cpu_manager.go:414] "containerMap: removing container" podUID="b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" containerName="istio-proxy" Oct 30 21:49:47.052063 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.052044 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" containerName="istio-proxy" Oct 30 21:49:47.052063 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.052053 5801 cpu_manager.go:414] "containerMap: removing container" podUID="8aa256c9-7e49-4144-9c43-91c8cdca8a7a" containerName="istio-proxy" Oct 30 21:49:47.052063 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.052058 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aa256c9-7e49-4144-9c43-91c8cdca8a7a" containerName="istio-proxy" Oct 30 21:49:47.052233 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.052073 5801 cpu_manager.go:414] "containerMap: removing container" podUID="4184fd8e-d9fb-474d-95b8-95afb19dac0b" containerName="istio-proxy" Oct 30 21:49:47.052233 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.052078 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4184fd8e-d9fb-474d-95b8-95afb19dac0b" containerName="istio-proxy" Oct 30 21:49:47.052233 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.052087 5801 cpu_manager.go:414] "containerMap: removing container" podUID="528a2150-a9c8-42d4-8203-113bc618287a" containerName="istio-proxy" Oct 30 21:49:47.052233 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.052091 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="528a2150-a9c8-42d4-8203-113bc618287a" containerName="istio-proxy" Oct 30 21:49:47.052233 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.052192 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aa256c9-7e49-4144-9c43-91c8cdca8a7a" containerName="istio-proxy" Oct 30 21:49:47.052233 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.052201 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" containerName="istio-proxy" Oct 30 21:49:47.052233 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.052210 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="4184fd8e-d9fb-474d-95b8-95afb19dac0b" containerName="istio-proxy" Oct 30 21:49:47.052233 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.052216 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="528a2150-a9c8-42d4-8203-113bc618287a" containerName="istio-proxy" Oct 30 21:49:47.115808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.112690 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-podinfo\") pod \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " Oct 30 21:49:47.115808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.112826 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-envoy\") pod \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " Oct 30 21:49:47.115808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.112953 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-data\") pod \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " Oct 30 21:49:47.115808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.113105 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istiod-ca-cert\") pod \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " Oct 30 21:49:47.115808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.113165 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-workload-socket\") pod \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " Oct 30 21:49:47.115808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.113205 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-credential-socket\") pod \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " Oct 30 21:49:47.115808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.113269 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4184fd8e-d9fb-474d-95b8-95afb19dac0b-kube-api-access-4fcvm\" (UniqueName: \"kubernetes.io/projected/4184fd8e-d9fb-474d-95b8-95afb19dac0b-kube-api-access-4fcvm\") pod \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " Oct 30 21:49:47.115808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.113440 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-token\" (UniqueName: \"kubernetes.io/projected/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-token\") pod \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " Oct 30 21:49:47.115808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.113577 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-workload-certs\") pod \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\" (UID: \"4184fd8e-d9fb-474d-95b8-95afb19dac0b\") " Oct 30 21:49:47.115808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.114490 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-workload-socket" pod "4184fd8e-d9fb-474d-95b8-95afb19dac0b" (UID: "4184fd8e-d9fb-474d-95b8-95afb19dac0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:47.115808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.115065 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:47.115808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.115372 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-workload-certs" pod "4184fd8e-d9fb-474d-95b8-95afb19dac0b" (UID: "4184fd8e-d9fb-474d-95b8-95afb19dac0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:47.115808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.115497 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-data" pod "4184fd8e-d9fb-474d-95b8-95afb19dac0b" (UID: "4184fd8e-d9fb-474d-95b8-95afb19dac0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:47.115808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.115628 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-credential-socket" pod "4184fd8e-d9fb-474d-95b8-95afb19dac0b" (UID: "4184fd8e-d9fb-474d-95b8-95afb19dac0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:47.120550 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.115899 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istiod-ca-cert" pod "4184fd8e-d9fb-474d-95b8-95afb19dac0b" (UID: "4184fd8e-d9fb-474d-95b8-95afb19dac0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:47.120550 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.116067 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.120550 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.118720 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-envoy" pod "4184fd8e-d9fb-474d-95b8-95afb19dac0b" (UID: "4184fd8e-d9fb-474d-95b8-95afb19dac0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:47.120550 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.118763 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-podinfo" pod "4184fd8e-d9fb-474d-95b8-95afb19dac0b" (UID: "4184fd8e-d9fb-474d-95b8-95afb19dac0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:49:47.121352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.120977 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-token" pod "4184fd8e-d9fb-474d-95b8-95afb19dac0b" (UID: "4184fd8e-d9fb-474d-95b8-95afb19dac0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:47.121352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.121248 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f8144d-openshift-default-dockercfg-b5qxh\"" Oct 30 21:49:47.121352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.121283 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4184fd8e-d9fb-474d-95b8-95afb19dac0b-kube-api-access-4fcvm" pod "4184fd8e-d9fb-474d-95b8-95afb19dac0b" (UID: "4184fd8e-d9fb-474d-95b8-95afb19dac0b"). InnerVolumeSpecName "kube-api-access-4fcvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:47.122973 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.122950 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf"] Oct 30 21:49:47.185333 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.184886 5801 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 21:49:47.185333 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.184947 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" podUID="409f33be-0503-4f36-ad79-93d1646f51ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 21:49:47.216776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.216720 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-data\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.216968 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.216878 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-credential-socket\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.217015 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.216962 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv727\" (UniqueName: \"kubernetes.io/projected/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-kube-api-access-fv727\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.217054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.217033 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-token\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.217105 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.217089 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-envoy\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.217149 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.217118 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istiod-ca-cert\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.217222 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.217203 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-workload-socket\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.217306 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.217295 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-podinfo\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.217344 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.217327 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-workload-certs\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.217436 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.217426 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:47.217471 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.217440 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:47.217471 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.217450 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:47.217471 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.217459 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fcvm\" (UniqueName: \"kubernetes.io/projected/4184fd8e-d9fb-474d-95b8-95afb19dac0b-kube-api-access-4fcvm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:47.217562 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.217470 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:47.217562 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.217481 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:47.217562 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.217489 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:47.217562 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.217498 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4184fd8e-d9fb-474d-95b8-95afb19dac0b-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:47.278093 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.277927 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps"] Oct 30 21:49:47.307193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.307106 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m98ps"] Oct 30 21:49:47.320800 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.318723 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-podinfo\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.320800 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.318817 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-workload-certs\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.320800 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.318927 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-data\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.320800 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.318999 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-credential-socket\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.320800 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.319060 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv727\" (UniqueName: \"kubernetes.io/projected/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-kube-api-access-fv727\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.320800 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.319146 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-token\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.320800 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.319208 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-envoy\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.320800 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.319244 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istiod-ca-cert\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.320800 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.319296 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-workload-socket\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.320800 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.319881 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-workload-socket\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.321546 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.321055 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-workload-certs\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.321546 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.321309 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-data\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.321660 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.321557 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-credential-socket\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.322602 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.322550 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istiod-ca-cert\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.323352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.323330 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-podinfo\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.324197 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.324127 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-envoy\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.324197 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.324162 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk"] Oct 30 21:49:47.330172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.330149 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-4hhlk"] Oct 30 21:49:47.334402 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.334379 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k"] Oct 30 21:49:47.335772 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.335733 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv727\" (UniqueName: \"kubernetes.io/projected/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-kube-api-access-fv727\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.336148 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.336128 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-token\") pod \"e2e-api-a-f8144d-openshift-default-5b96d887-msbqf\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.338175 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.338157 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-qcw6k"] Oct 30 21:49:47.460141 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.459828 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:47.536082 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.533114 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:47.545249 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.545221 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="528a2150-a9c8-42d4-8203-113bc618287a" path="/var/lib/kubelet/pods/528a2150-a9c8-42d4-8203-113bc618287a/volumes" Oct 30 21:49:47.546038 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.546016 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aa256c9-7e49-4144-9c43-91c8cdca8a7a" path="/var/lib/kubelet/pods/8aa256c9-7e49-4144-9c43-91c8cdca8a7a/volumes" Oct 30 21:49:47.546725 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.546708 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7" path="/var/lib/kubelet/pods/b88c55fe-afa3-4b4e-b5b7-e4f8e35e51d7/volumes" Oct 30 21:49:47.681775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.677000 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:47.752058 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.751964 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" Oct 30 21:49:47.752444 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.752018 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9" event={"ID":"4184fd8e-d9fb-474d-95b8-95afb19dac0b","Type":"ContainerDied","Data":"f0c3459116a6fcb79b4a177486dd83b2799524027e7d21b7d28ae76500be5861"} Oct 30 21:49:47.752705 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.752678 5801 scope.go:122] "RemoveContainer" containerID="ad1c82ca03e906c32be1c4c0c5ef5f134cde5d73c9022e761ae7dedd5038f81b" Oct 30 21:49:47.851777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.821547 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:47.911892 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.911851 5801 patch_prober.go:29] interesting pod/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 21:49:47.912166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.912134 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" podUID="f9975b4c-127e-4130-ae64-b8785b0acc7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 21:49:47.990776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.978791 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9"] Oct 30 21:49:47.990776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.978868 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7dfff0-openshift-default-559bb574c5-m6xq9"] Oct 30 21:49:47.990776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.978895 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb"] Oct 30 21:49:48.012776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:47.994933 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:48.016793 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:48.014194 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-000f64-q44pb"] Oct 30 21:49:48.175294 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:48.175256 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4" podUID="951263d7-9273-48d6-9f89-e60bf70b7998" containerName="payload-processing" containerID="cri-o://5a1829a60e66a5e11864e107b300811366745ec867acf01c1f734c7a69b2d330" gracePeriod=25 Oct 30 21:49:48.176779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:48.176171 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7" podUID="9d03a2c0-4527-4104-ac16-dbadd1b59a97" containerName="payload-pre-processing" containerID="cri-o://033820a601dfbf7b22f31904915e2d391d22452c7ffae1573a4dd87d2153b601" gracePeriod=25 Oct 30 21:49:48.177429 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:48.177406 5801 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 21:49:48.177571 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:48.177551 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" podUID="409f33be-0503-4f36-ad79-93d1646f51ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 21:49:48.415132 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:48.415004 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:48.536921 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:48.524390 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:48.546397 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:48.538719 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf"] Oct 30 21:49:48.580787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:48.577215 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:49:48.580787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:48.577309 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:49:48.580787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:48.577353 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:49:48.838432 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:48.815600 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:48.944822 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:48.941358 5801 patch_prober.go:29] interesting pod/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 21:49:48.944822 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:48.941417 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" podUID="f9975b4c-127e-4130-ae64-b8785b0acc7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 21:49:49.213180 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:49.212732 5801 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 21:49:49.213180 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:49.212808 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" podUID="409f33be-0503-4f36-ad79-93d1646f51ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 21:49:49.219209 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:49.218642 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" event={"ID":"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4","Type":"ContainerStarted","Data":"c93df807ae8964af71a3409aa6a1648256ff8e7c83955465ddb73e28e4ad28a0"} Oct 30 21:49:49.224330 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:49.224302 5801 generic.go:356] "Generic (PLEG): container finished" podID="951263d7-9273-48d6-9f89-e60bf70b7998" containerID="5a1829a60e66a5e11864e107b300811366745ec867acf01c1f734c7a69b2d330" exitCode=0 Oct 30 21:49:49.225293 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:49.224515 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4" event={"ID":"951263d7-9273-48d6-9f89-e60bf70b7998","Type":"ContainerDied","Data":"5a1829a60e66a5e11864e107b300811366745ec867acf01c1f734c7a69b2d330"} Oct 30 21:49:49.228162 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:49.227635 5801 generic.go:356] "Generic (PLEG): container finished" podID="9d03a2c0-4527-4104-ac16-dbadd1b59a97" containerID="033820a601dfbf7b22f31904915e2d391d22452c7ffae1573a4dd87d2153b601" exitCode=0 Oct 30 21:49:49.228162 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:49.227745 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7" event={"ID":"9d03a2c0-4527-4104-ac16-dbadd1b59a97","Type":"ContainerDied","Data":"033820a601dfbf7b22f31904915e2d391d22452c7ffae1573a4dd87d2153b601"} Oct 30 21:49:49.826442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:49.826410 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:49.837736 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:49.837704 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4184fd8e-d9fb-474d-95b8-95afb19dac0b" path="/var/lib/kubelet/pods/4184fd8e-d9fb-474d-95b8-95afb19dac0b/volumes" Oct 30 21:49:49.838483 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:49.838462 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9" path="/var/lib/kubelet/pods/9fb45b98-bff7-4fc8-93b3-bb68fb9c2ed9/volumes" Oct 30 21:49:49.840773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:49.838926 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-950784-7rs77"] Oct 30 21:49:49.840773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:49.840399 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aa256c9-7e49-4144-9c43-91c8cdca8a7a" containerName="istio-proxy" Oct 30 21:49:49.845980 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:49.845960 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-950784-7rs77"] Oct 30 21:49:49.846085 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:49.846073 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-950784-7rs77" Oct 30 21:49:49.849532 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:49.849512 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:49:49.850620 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:49.850604 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:49:49.917944 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:49.917908 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:49:49.918306 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:49.918284 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:49:49.919062 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:49.919035 5801 patch_prober.go:29] interesting pod/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 21:49:49.919280 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:49.919251 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" podUID="f9975b4c-127e-4130-ae64-b8785b0acc7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 21:49:49.924900 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:49.924709 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4" Oct 30 21:49:49.929401 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:49.929381 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7" Oct 30 21:49:50.030632 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.030530 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/951263d7-9273-48d6-9f89-e60bf70b7998-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951263d7-9273-48d6-9f89-e60bf70b7998-plugins-config-volume\") pod \"951263d7-9273-48d6-9f89-e60bf70b7998\" (UID: \"951263d7-9273-48d6-9f89-e60bf70b7998\") " Oct 30 21:49:50.030854 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.030647 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d03a2c0-4527-4104-ac16-dbadd1b59a97-kube-api-access-4nzh6\" (UniqueName: \"kubernetes.io/projected/9d03a2c0-4527-4104-ac16-dbadd1b59a97-kube-api-access-4nzh6\") pod \"9d03a2c0-4527-4104-ac16-dbadd1b59a97\" (UID: \"9d03a2c0-4527-4104-ac16-dbadd1b59a97\") " Oct 30 21:49:50.030943 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.030880 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d03a2c0-4527-4104-ac16-dbadd1b59a97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d03a2c0-4527-4104-ac16-dbadd1b59a97-plugins-config-volume\") pod \"9d03a2c0-4527-4104-ac16-dbadd1b59a97\" (UID: \"9d03a2c0-4527-4104-ac16-dbadd1b59a97\") " Oct 30 21:49:50.031002 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.030957 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/951263d7-9273-48d6-9f89-e60bf70b7998-kube-api-access-dtmm7\" (UniqueName: \"kubernetes.io/projected/951263d7-9273-48d6-9f89-e60bf70b7998-kube-api-access-dtmm7\") pod \"951263d7-9273-48d6-9f89-e60bf70b7998\" (UID: \"951263d7-9273-48d6-9f89-e60bf70b7998\") " Oct 30 21:49:50.031002 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.030962 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/951263d7-9273-48d6-9f89-e60bf70b7998-plugins-config-volume" pod "951263d7-9273-48d6-9f89-e60bf70b7998" (UID: "951263d7-9273-48d6-9f89-e60bf70b7998"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:50.031406 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.031376 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/def78ea4-463d-4580-9a63-c1f9be5507f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-950784-7rs77\" (UID: \"def78ea4-463d-4580-9a63-c1f9be5507f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-950784-7rs77" Oct 30 21:49:50.031510 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.031481 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d03a2c0-4527-4104-ac16-dbadd1b59a97-plugins-config-volume" pod "9d03a2c0-4527-4104-ac16-dbadd1b59a97" (UID: "9d03a2c0-4527-4104-ac16-dbadd1b59a97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:50.031592 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.031560 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrqkh\" (UniqueName: \"kubernetes.io/projected/def78ea4-463d-4580-9a63-c1f9be5507f9-kube-api-access-nrqkh\") pod \"maas-api-revoke-keys-e2e-api-b-950784-7rs77\" (UID: \"def78ea4-463d-4580-9a63-c1f9be5507f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-950784-7rs77" Oct 30 21:49:50.031654 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.031602 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/def78ea4-463d-4580-9a63-c1f9be5507f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-950784-7rs77\" (UID: \"def78ea4-463d-4580-9a63-c1f9be5507f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-950784-7rs77" Oct 30 21:49:50.031985 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.031966 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951263d7-9273-48d6-9f89-e60bf70b7998-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:50.032117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.032103 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d03a2c0-4527-4104-ac16-dbadd1b59a97-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:50.033117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.033097 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d03a2c0-4527-4104-ac16-dbadd1b59a97-kube-api-access-4nzh6" pod "9d03a2c0-4527-4104-ac16-dbadd1b59a97" (UID: "9d03a2c0-4527-4104-ac16-dbadd1b59a97"). InnerVolumeSpecName "kube-api-access-4nzh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:50.033604 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.033582 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951263d7-9273-48d6-9f89-e60bf70b7998-kube-api-access-dtmm7" pod "951263d7-9273-48d6-9f89-e60bf70b7998" (UID: "951263d7-9273-48d6-9f89-e60bf70b7998"). InnerVolumeSpecName "kube-api-access-dtmm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:50.133567 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.133532 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrqkh\" (UniqueName: \"kubernetes.io/projected/def78ea4-463d-4580-9a63-c1f9be5507f9-kube-api-access-nrqkh\") pod \"maas-api-revoke-keys-e2e-api-b-950784-7rs77\" (UID: \"def78ea4-463d-4580-9a63-c1f9be5507f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-950784-7rs77" Oct 30 21:49:50.133819 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.133600 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/def78ea4-463d-4580-9a63-c1f9be5507f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-950784-7rs77\" (UID: \"def78ea4-463d-4580-9a63-c1f9be5507f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-950784-7rs77" Oct 30 21:49:50.133906 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.133874 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/def78ea4-463d-4580-9a63-c1f9be5507f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-950784-7rs77\" (UID: \"def78ea4-463d-4580-9a63-c1f9be5507f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-950784-7rs77" Oct 30 21:49:50.133997 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.133974 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nzh6\" (UniqueName: \"kubernetes.io/projected/9d03a2c0-4527-4104-ac16-dbadd1b59a97-kube-api-access-4nzh6\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:50.134058 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.134001 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtmm7\" (UniqueName: \"kubernetes.io/projected/951263d7-9273-48d6-9f89-e60bf70b7998-kube-api-access-dtmm7\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:50.134665 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.134646 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/def78ea4-463d-4580-9a63-c1f9be5507f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-950784-7rs77\" (UID: \"def78ea4-463d-4580-9a63-c1f9be5507f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-950784-7rs77" Oct 30 21:49:50.136138 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.136119 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/def78ea4-463d-4580-9a63-c1f9be5507f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-950784-7rs77\" (UID: \"def78ea4-463d-4580-9a63-c1f9be5507f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-950784-7rs77" Oct 30 21:49:50.145833 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.145815 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrqkh\" (UniqueName: \"kubernetes.io/projected/def78ea4-463d-4580-9a63-c1f9be5507f9-kube-api-access-nrqkh\") pod \"maas-api-revoke-keys-e2e-api-b-950784-7rs77\" (UID: \"def78ea4-463d-4580-9a63-c1f9be5507f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-950784-7rs77" Oct 30 21:49:50.191807 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.182788 5801 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 21:49:50.191807 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.182852 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" podUID="409f33be-0503-4f36-ad79-93d1646f51ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 21:49:50.239828 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.239788 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" event={"ID":"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4","Type":"ContainerStarted","Data":"6200424ff91b7331861a8103fd76e92a79a228b40526b3f653f80e377e873dcf"} Oct 30 21:49:50.240287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.240254 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-950784-7rs77" Oct 30 21:49:50.241364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.241346 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4" Oct 30 21:49:50.241630 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.241352 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4" event={"ID":"951263d7-9273-48d6-9f89-e60bf70b7998","Type":"ContainerDied","Data":"aaefe7c4fa60d7afb22636704775d2505e7cbd7f4e1a86ff8d300e3038851afd"} Oct 30 21:49:50.241697 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.241671 5801 scope.go:122] "RemoveContainer" containerID="5a1829a60e66a5e11864e107b300811366745ec867acf01c1f734c7a69b2d330" Oct 30 21:49:50.243036 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.242646 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7" event={"ID":"9d03a2c0-4527-4104-ac16-dbadd1b59a97","Type":"ContainerDied","Data":"d57a282fe7e388f1d49eb3e50e24b8e4afe6a572697f1b563ce8e64f97dbd0a0"} Oct 30 21:49:50.243036 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.242658 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7" Oct 30 21:49:50.325859 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.325806 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" podStartSLOduration=3.325735053 podStartE2EDuration="3.325735053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:50.290527101 +0000 UTC m=+1258.886343773" watchObservedRunningTime="2026-10-30 21:49:50.325735053 +0000 UTC m=+1258.921551712" Oct 30 21:49:50.328491 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.328472 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4"] Oct 30 21:49:50.328570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.328499 5801 scope.go:122] "RemoveContainer" containerID="033820a601dfbf7b22f31904915e2d391d22452c7ffae1573a4dd87d2153b601" Oct 30 21:49:50.334287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.334261 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4"] Oct 30 21:49:50.346854 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.346499 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7"] Oct 30 21:49:50.351264 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.351237 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7"] Oct 30 21:49:50.467361 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.467328 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:50.467505 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.467457 5801 patch_prober.go:29] interesting pod/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Oct 30 21:49:50.467585 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.467497 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" podUID="0e91a97c-e981-4ab6-b6cd-3f92381ed5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Oct 30 21:49:50.643718 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.643577 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:50.905686 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.905593 5801 patch_prober.go:29] interesting pod/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 21:49:50.922624 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.922581 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" podUID="f9975b4c-127e-4130-ae64-b8785b0acc7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 21:49:50.931470 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.924633 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-b-2ba0bd-8596664f99-sfwk4" podUID="951263d7-9273-48d6-9f89-e60bf70b7998" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.182:50052\" within 1s: context deadline exceeded" Oct 30 21:49:50.943726 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.943686 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ba0bd-674b49c55-5lth7" podUID="9d03a2c0-4527-4104-ac16-dbadd1b59a97" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.181:50052\" within 1s: context deadline exceeded" Oct 30 21:49:50.944112 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:49:50.944092 5801 logging.go:55] [core] [Channel #1901 SubChannel #1904]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.181:50052", ServerName: "10.133.0.181:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.181:50052: operation was canceled" Oct 30 21:49:50.944329 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:49:50.944316 5801 logging.go:55] [core] [Channel #1902 SubChannel #1903]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.182:50052", ServerName: "10.133.0.182:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.182:50052: operation was canceled" Oct 30 21:49:50.956805 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:50.952676 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-950784-7rs77"] Oct 30 21:49:51.165428 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:51.164149 5801 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 21:49:51.165428 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:51.164212 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" podUID="409f33be-0503-4f36-ad79-93d1646f51ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 21:49:51.304776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:51.304133 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-950784-7rs77" event={"ID":"def78ea4-463d-4580-9a63-c1f9be5507f9","Type":"ContainerStarted","Data":"052a89070e8fcc51e6349a43b43ceb92a30e1182b5cc391eca41095b370d9440"} Oct 30 21:49:51.470394 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:51.470303 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:51.472878 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:51.472848 5801 patch_prober.go:29] interesting pod/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Oct 30 21:49:51.473097 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:51.473071 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" podUID="0e91a97c-e981-4ab6-b6cd-3f92381ed5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Oct 30 21:49:51.543257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:51.543218 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="951263d7-9273-48d6-9f89-e60bf70b7998" path="/var/lib/kubelet/pods/951263d7-9273-48d6-9f89-e60bf70b7998/volumes" Oct 30 21:49:51.544156 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:51.544128 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d03a2c0-4527-4104-ac16-dbadd1b59a97" path="/var/lib/kubelet/pods/9d03a2c0-4527-4104-ac16-dbadd1b59a97/volumes" Oct 30 21:49:51.902049 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:51.902022 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:49:51.902512 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:51.902498 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:49:51.903287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:51.903267 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:49:52.144364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:52.144329 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:52.186875 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:52.184352 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:52.186875 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:52.185168 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:52.187364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:52.187342 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:49:52.323475 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:52.323439 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-950784-7rs77" event={"ID":"def78ea4-463d-4580-9a63-c1f9be5507f9","Type":"ContainerStarted","Data":"f290e20df2e40cf53b11a519ac4cfcc746d6ab75d31f3cdf4778ea84a262829d"} Oct 30 21:49:52.390161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:52.387842 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-950784-7rs77" podStartSLOduration=3.387798375 podStartE2EDuration="3.387798375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:52.353025678 +0000 UTC m=+1260.948842366" watchObservedRunningTime="2026-10-30 21:49:52.387798375 +0000 UTC m=+1260.983615035" Oct 30 21:49:52.480112 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:52.480017 5801 patch_prober.go:29] interesting pod/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Oct 30 21:49:52.480112 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:52.480080 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" podUID="0e91a97c-e981-4ab6-b6cd-3f92381ed5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Oct 30 21:49:52.829252 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:52.829214 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:53.332546 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:53.332510 5801 generic.go:356] "Generic (PLEG): container finished" podID="def78ea4-463d-4580-9a63-c1f9be5507f9" containerID="f290e20df2e40cf53b11a519ac4cfcc746d6ab75d31f3cdf4778ea84a262829d" exitCode=0 Oct 30 21:49:53.333093 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:53.332561 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-950784-7rs77" event={"ID":"def78ea4-463d-4580-9a63-c1f9be5507f9","Type":"ContainerDied","Data":"f290e20df2e40cf53b11a519ac4cfcc746d6ab75d31f3cdf4778ea84a262829d"} Oct 30 21:49:53.479001 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:53.478962 5801 patch_prober.go:29] interesting pod/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Oct 30 21:49:53.479295 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:53.479259 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" podUID="0e91a97c-e981-4ab6-b6cd-3f92381ed5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Oct 30 21:49:53.574775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:53.573012 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:54.279447 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:54.279408 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:54.401709 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:54.401669 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84" Oct 30 21:49:54.404026 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:54.403595 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9" Oct 30 21:49:54.496044 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:54.495443 5801 patch_prober.go:29] interesting pod/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Oct 30 21:49:54.496044 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:54.495500 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" podUID="0e91a97c-e981-4ab6-b6cd-3f92381ed5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Oct 30 21:49:54.695016 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:54.694981 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-950784-7rs77" Oct 30 21:49:54.798700 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:54.798668 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def78ea4-463d-4580-9a63-c1f9be5507f9-kube-api-access-nrqkh\" (UniqueName: \"kubernetes.io/projected/def78ea4-463d-4580-9a63-c1f9be5507f9-kube-api-access-nrqkh\") pod \"def78ea4-463d-4580-9a63-c1f9be5507f9\" (UID: \"def78ea4-463d-4580-9a63-c1f9be5507f9\") " Oct 30 21:49:54.798906 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:54.798785 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/def78ea4-463d-4580-9a63-c1f9be5507f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/def78ea4-463d-4580-9a63-c1f9be5507f9-maas-api-openshift-ca\") pod \"def78ea4-463d-4580-9a63-c1f9be5507f9\" (UID: \"def78ea4-463d-4580-9a63-c1f9be5507f9\") " Oct 30 21:49:54.798956 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:54.798920 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/def78ea4-463d-4580-9a63-c1f9be5507f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/def78ea4-463d-4580-9a63-c1f9be5507f9-maas-api-xks-ca\") pod \"def78ea4-463d-4580-9a63-c1f9be5507f9\" (UID: \"def78ea4-463d-4580-9a63-c1f9be5507f9\") " Oct 30 21:49:54.799088 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:54.799064 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def78ea4-463d-4580-9a63-c1f9be5507f9-maas-api-openshift-ca" pod "def78ea4-463d-4580-9a63-c1f9be5507f9" (UID: "def78ea4-463d-4580-9a63-c1f9be5507f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:54.799650 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:54.799626 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/def78ea4-463d-4580-9a63-c1f9be5507f9-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:54.800927 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:54.800899 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def78ea4-463d-4580-9a63-c1f9be5507f9-maas-api-xks-ca" pod "def78ea4-463d-4580-9a63-c1f9be5507f9" (UID: "def78ea4-463d-4580-9a63-c1f9be5507f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:54.801050 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:54.800985 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def78ea4-463d-4580-9a63-c1f9be5507f9-kube-api-access-nrqkh" pod "def78ea4-463d-4580-9a63-c1f9be5507f9" (UID: "def78ea4-463d-4580-9a63-c1f9be5507f9"). InnerVolumeSpecName "kube-api-access-nrqkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:54.901203 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:54.901125 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrqkh\" (UniqueName: \"kubernetes.io/projected/def78ea4-463d-4580-9a63-c1f9be5507f9-kube-api-access-nrqkh\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:54.901203 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:54.901163 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/def78ea4-463d-4580-9a63-c1f9be5507f9-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:55.185872 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:55.184952 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:55.240040 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:55.240005 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k"] Oct 30 21:49:55.240251 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:55.240231 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" podUID="b99efd6a-6f7d-444a-89dd-bf292349a7cc" containerName="istio-proxy" containerID="cri-o://1bf31569b29eaa967754529bab060ef16dc67f490f8903d46b73b29628056e28" gracePeriod=30 Oct 30 21:49:55.282816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:55.282764 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:55.481865 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:55.480919 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-950784-7rs77" event={"ID":"def78ea4-463d-4580-9a63-c1f9be5507f9","Type":"ContainerDied","Data":"052a89070e8fcc51e6349a43b43ceb92a30e1182b5cc391eca41095b370d9440"} Oct 30 21:49:55.481865 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:55.480988 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="052a89070e8fcc51e6349a43b43ceb92a30e1182b5cc391eca41095b370d9440" Oct 30 21:49:55.481865 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:55.481079 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-950784-7rs77" Oct 30 21:49:55.540562 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:55.540465 5801 patch_prober.go:29] interesting pod/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Oct 30 21:49:55.540562 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:55.540527 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" podUID="0e91a97c-e981-4ab6-b6cd-3f92381ed5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Oct 30 21:49:55.634802 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:55.626528 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:56.028865 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:56.028826 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:56.211834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:56.211794 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:56.321993 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:56.320045 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:56.490898 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:56.490864 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:56.492238 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:56.492219 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:56.493079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:56.493061 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:49:56.655229 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:56.655122 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:56.758625 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:56.758541 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:57.004038 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:57.003957 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm"] Oct 30 21:49:57.004455 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:57.004424 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" podUID="9ea6c627-8885-490d-88ee-10befd58f673" containerName="istio-proxy" containerID="cri-o://a11e2faeda89f468c8c762c0506a20872a9c71771fa904f821982ddf4cf4501e" gracePeriod=30 Oct 30 21:49:57.031708 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:57.031657 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:57.138452 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:57.138414 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:57.313974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:57.308773 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:57.717859 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:57.717772 5801 scope.go:122] "RemoveContainer" containerID="985eb575ae8aac117e9234cf0f862713e22b28434929f5e5cd9b558dcdbf4fdd" Oct 30 21:49:57.729004 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:57.728979 5801 scope.go:122] "RemoveContainer" containerID="31ec94472f02bd232bed28a224d01a997dc2edcc5ee0bbfea94533e23ec3783c" Oct 30 21:49:57.739114 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:57.739093 5801 scope.go:122] "RemoveContainer" containerID="638fe8a80408674f12e39639b98ffe18374d659bb6e499a226edfad31de8288c" Oct 30 21:49:57.749982 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:57.749960 5801 scope.go:122] "RemoveContainer" containerID="5efd3820b797c651ae5a9107a42651fc8530aebb86603a66f3f148a2f40de5b4" Oct 30 21:49:58.040372 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:58.040283 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:58.145300 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:58.145263 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:58.835998 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:58.835966 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v"] Oct 30 21:49:58.836874 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:58.836853 5801 cpu_manager.go:414] "containerMap: removing container" podUID="def78ea4-463d-4580-9a63-c1f9be5507f9" containerName="revoke-keys" Oct 30 21:49:58.836994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:58.836874 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def78ea4-463d-4580-9a63-c1f9be5507f9" containerName="revoke-keys" Oct 30 21:49:58.836994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:58.836939 5801 cpu_manager.go:414] "containerMap: removing container" podUID="9d03a2c0-4527-4104-ac16-dbadd1b59a97" containerName="payload-pre-processing" Oct 30 21:49:58.836994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:58.836948 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d03a2c0-4527-4104-ac16-dbadd1b59a97" containerName="payload-pre-processing" Oct 30 21:49:58.836994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:58.836974 5801 cpu_manager.go:414] "containerMap: removing container" podUID="951263d7-9273-48d6-9f89-e60bf70b7998" containerName="payload-processing" Oct 30 21:49:58.836994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:58.836993 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="951263d7-9273-48d6-9f89-e60bf70b7998" containerName="payload-processing" Oct 30 21:49:58.837258 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:58.837174 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d03a2c0-4527-4104-ac16-dbadd1b59a97" containerName="payload-pre-processing" Oct 30 21:49:58.837258 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:58.837190 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="951263d7-9273-48d6-9f89-e60bf70b7998" containerName="payload-processing" Oct 30 21:49:58.837258 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:58.837207 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="def78ea4-463d-4580-9a63-c1f9be5507f9" containerName="revoke-keys" Oct 30 21:49:58.847695 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:58.845108 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:58.862614 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:58.859287 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v"] Oct 30 21:49:58.887798 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:58.886625 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:59.018332 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.018291 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-credential-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.018332 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.018336 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gg7r\" (UniqueName: \"kubernetes.io/projected/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-kube-api-access-5gg7r\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.018677 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.018367 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-data\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.018677 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.018488 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-token\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.018677 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.018509 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-envoy\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.018677 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.018557 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-workload-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.018944 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.018685 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istiod-ca-cert\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.018944 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.018711 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-workload-certs\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.018944 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.018730 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-podinfo\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.120156 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.120057 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-workload-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.120324 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.120265 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istiod-ca-cert\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.120324 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.120306 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-workload-certs\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.120457 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.120339 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-podinfo\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.120520 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.120459 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-credential-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.120520 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.120477 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-workload-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.120520 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.120490 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gg7r\" (UniqueName: \"kubernetes.io/projected/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-kube-api-access-5gg7r\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.120728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.120545 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-data\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.120728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.120618 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-token\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.120728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.120652 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-envoy\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.120728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.120720 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-credential-socket\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.121008 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.120652 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-workload-certs\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.121008 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.120905 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-data\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.121129 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.121066 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istiod-ca-cert\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.122894 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.122869 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-envoy\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.123037 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.123016 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-podinfo\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.138499 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.138469 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-token\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.138646 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.138598 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gg7r\" (UniqueName: \"kubernetes.io/projected/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-kube-api-access-5gg7r\") pod \"e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.170687 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.170652 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:49:59.479241 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.479208 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v"] Oct 30 21:49:59.479954 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:49:59.479929 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70abeac9_ada5_4b73_8242_5fb3cad8e5a6.slice/crio-c677b155562a5c913f5d0698df7942d976434d7e35a5c446ebe1f308b37d30ee WatchSource:0}: Error finding container c677b155562a5c913f5d0698df7942d976434d7e35a5c446ebe1f308b37d30ee: Status 404 returned error can't find the container with id c677b155562a5c913f5d0698df7942d976434d7e35a5c446ebe1f308b37d30ee Oct 30 21:49:59.483190 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.483157 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:49:59.483298 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.483224 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:49:59.483298 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.483256 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:49:59.526338 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.526309 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" event={"ID":"70abeac9-ada5-4b73-8242-5fb3cad8e5a6","Type":"ContainerStarted","Data":"c677b155562a5c913f5d0698df7942d976434d7e35a5c446ebe1f308b37d30ee"} Oct 30 21:49:59.679729 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.679692 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz"] Oct 30 21:49:59.697249 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.697209 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6"] Oct 30 21:49:59.777216 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.777122 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:59.881054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.881014 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:59.977867 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:49:59.977828 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:00.007646 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.007610 5801 patch_prober.go:29] interesting pod/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:50:00.007850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.007689 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" podUID="9ea6c627-8885-490d-88ee-10befd58f673" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:50:00.170775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.169167 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:00.246778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.240804 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x"] Oct 30 21:50:00.280775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.272483 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ba0bd-k582x"] Oct 30 21:50:00.280775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.279551 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:00.528975 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:50:00.519150 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:50:00.533568 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.533535 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:00.568794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.553239 5801 generic.go:356] "Generic (PLEG): container finished" podID="b99efd6a-6f7d-444a-89dd-bf292349a7cc" containerID="1bf31569b29eaa967754529bab060ef16dc67f490f8903d46b73b29628056e28" exitCode=0 Oct 30 21:50:00.568794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.553400 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" event={"ID":"b99efd6a-6f7d-444a-89dd-bf292349a7cc","Type":"ContainerDied","Data":"1bf31569b29eaa967754529bab060ef16dc67f490f8903d46b73b29628056e28"} Oct 30 21:50:00.613817 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.612796 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" event={"ID":"70abeac9-ada5-4b73-8242-5fb3cad8e5a6","Type":"ContainerStarted","Data":"8e69e57bf3f8c4647ad767fddd664826b57b23600102c0cd5c9a787f512f920a"} Oct 30 21:50:00.715212 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.715142 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" podStartSLOduration=2.7151179279999997 podStartE2EDuration="2.715117928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:00.70650003 +0000 UTC m=+1269.302316717" watchObservedRunningTime="2026-10-30 21:50:00.715117928 +0000 UTC m=+1269.310934589" Oct 30 21:50:00.738315 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.738280 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:50:00.821612 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.795541 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:00.920512 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.920479 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-workload-certs\") pod \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " Oct 30 21:50:00.921029 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.920531 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-token\" (UniqueName: \"kubernetes.io/projected/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-token\") pod \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " Oct 30 21:50:00.921029 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.920581 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-data\") pod \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " Oct 30 21:50:00.921029 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.920698 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istiod-ca-cert\") pod \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " Oct 30 21:50:00.921029 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.920725 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-envoy\") pod \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " Oct 30 21:50:00.921029 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.920782 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b99efd6a-6f7d-444a-89dd-bf292349a7cc-kube-api-access-5cddz\" (UniqueName: \"kubernetes.io/projected/b99efd6a-6f7d-444a-89dd-bf292349a7cc-kube-api-access-5cddz\") pod \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " Oct 30 21:50:00.921029 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.920808 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-workload-certs" pod "b99efd6a-6f7d-444a-89dd-bf292349a7cc" (UID: "b99efd6a-6f7d-444a-89dd-bf292349a7cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:00.921029 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.920824 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-credential-socket\") pod \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " Oct 30 21:50:00.921029 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.920854 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-podinfo\") pod \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " Oct 30 21:50:00.921029 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.920892 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-workload-socket\") pod \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\" (UID: \"b99efd6a-6f7d-444a-89dd-bf292349a7cc\") " Oct 30 21:50:00.921029 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.920997 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-data" pod "b99efd6a-6f7d-444a-89dd-bf292349a7cc" (UID: "b99efd6a-6f7d-444a-89dd-bf292349a7cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:00.921728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.921144 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-credential-socket" pod "b99efd6a-6f7d-444a-89dd-bf292349a7cc" (UID: "b99efd6a-6f7d-444a-89dd-bf292349a7cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:00.921728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.921106 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istiod-ca-cert" pod "b99efd6a-6f7d-444a-89dd-bf292349a7cc" (UID: "b99efd6a-6f7d-444a-89dd-bf292349a7cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:00.921728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.921297 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-workload-socket" pod "b99efd6a-6f7d-444a-89dd-bf292349a7cc" (UID: "b99efd6a-6f7d-444a-89dd-bf292349a7cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:00.921728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.921666 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:00.921728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.921689 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:00.921728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.921705 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:00.921728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.921719 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:00.921728 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.921733 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:00.923082 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.923055 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-envoy" pod "b99efd6a-6f7d-444a-89dd-bf292349a7cc" (UID: "b99efd6a-6f7d-444a-89dd-bf292349a7cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:00.923178 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.923145 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-token" pod "b99efd6a-6f7d-444a-89dd-bf292349a7cc" (UID: "b99efd6a-6f7d-444a-89dd-bf292349a7cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:00.923234 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.923187 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99efd6a-6f7d-444a-89dd-bf292349a7cc-kube-api-access-5cddz" pod "b99efd6a-6f7d-444a-89dd-bf292349a7cc" (UID: "b99efd6a-6f7d-444a-89dd-bf292349a7cc"). InnerVolumeSpecName "kube-api-access-5cddz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:00.923296 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:00.923278 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-podinfo" pod "b99efd6a-6f7d-444a-89dd-bf292349a7cc" (UID: "b99efd6a-6f7d-444a-89dd-bf292349a7cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:01.023442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:01.023403 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:01.023442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:01.023438 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:01.023442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:01.023451 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cddz\" (UniqueName: \"kubernetes.io/projected/b99efd6a-6f7d-444a-89dd-bf292349a7cc-kube-api-access-5cddz\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:01.023701 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:01.023460 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99efd6a-6f7d-444a-89dd-bf292349a7cc-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:01.179019 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:01.178919 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:50:01.179539 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:01.179508 5801 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 21:50:01.179659 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:01.179562 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" podUID="70abeac9-ada5-4b73-8242-5fb3cad8e5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 21:50:01.484234 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:01.484138 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:01.589450 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:01.589413 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ba41b66-8be4-4424-a326-c7e86dc1ffcf" path="/var/lib/kubelet/pods/3ba41b66-8be4-4424-a326-c7e86dc1ffcf/volumes" Oct 30 21:50:01.623248 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:01.623219 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" Oct 30 21:50:01.623248 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:01.623237 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k" event={"ID":"b99efd6a-6f7d-444a-89dd-bf292349a7cc","Type":"ContainerDied","Data":"320927bfb46c5b02dd5f0df6648d8b1171cdcbdde4bee7fd24000f6132bd70cc"} Oct 30 21:50:01.623489 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:01.623297 5801 scope.go:122] "RemoveContainer" containerID="1bf31569b29eaa967754529bab060ef16dc67f490f8903d46b73b29628056e28" Oct 30 21:50:01.648927 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:01.648900 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k"] Oct 30 21:50:01.678776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:01.678663 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-n876k"] Oct 30 21:50:02.173776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.171908 5801 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 21:50:02.173776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.171988 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" podUID="70abeac9-ada5-4b73-8242-5fb3cad8e5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 21:50:02.379734 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.379597 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:02.645305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.645221 5801 generic.go:356] "Generic (PLEG): container finished" podID="9ea6c627-8885-490d-88ee-10befd58f673" containerID="a11e2faeda89f468c8c762c0506a20872a9c71771fa904f821982ddf4cf4501e" exitCode=0 Oct 30 21:50:02.645305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.645290 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" event={"ID":"9ea6c627-8885-490d-88ee-10befd58f673","Type":"ContainerDied","Data":"a11e2faeda89f468c8c762c0506a20872a9c71771fa904f821982ddf4cf4501e"} Oct 30 21:50:02.645549 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.645350 5801 scope.go:122] "RemoveContainer" containerID="26f388b059480b1f1ecc6f86cea4c809d8d4230370c30f25da41e897eca4993a" Oct 30 21:50:02.819827 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.819800 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:50:02.897226 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.897116 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ea6c627-8885-490d-88ee-10befd58f673-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ea6c627-8885-490d-88ee-10befd58f673-istiod-ca-cert\") pod \"9ea6c627-8885-490d-88ee-10befd58f673\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " Oct 30 21:50:02.897226 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.897212 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-istio-envoy\") pod \"9ea6c627-8885-490d-88ee-10befd58f673\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " Oct 30 21:50:02.897463 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.897245 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ea6c627-8885-490d-88ee-10befd58f673-istio-token\" (UniqueName: \"kubernetes.io/projected/9ea6c627-8885-490d-88ee-10befd58f673-istio-token\") pod \"9ea6c627-8885-490d-88ee-10befd58f673\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " Oct 30 21:50:02.897463 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.897428 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-workload-socket\") pod \"9ea6c627-8885-490d-88ee-10befd58f673\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " Oct 30 21:50:02.897590 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.897462 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-credential-socket\") pod \"9ea6c627-8885-490d-88ee-10befd58f673\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " Oct 30 21:50:02.897590 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.897534 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ea6c627-8885-490d-88ee-10befd58f673-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ea6c627-8885-490d-88ee-10befd58f673-istio-podinfo\") pod \"9ea6c627-8885-490d-88ee-10befd58f673\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " Oct 30 21:50:02.897711 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.897687 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ea6c627-8885-490d-88ee-10befd58f673-kube-api-access-8tl2b\" (UniqueName: \"kubernetes.io/projected/9ea6c627-8885-490d-88ee-10befd58f673-kube-api-access-8tl2b\") pod \"9ea6c627-8885-490d-88ee-10befd58f673\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " Oct 30 21:50:02.897789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.897742 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-workload-certs\") pod \"9ea6c627-8885-490d-88ee-10befd58f673\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " Oct 30 21:50:02.897864 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.897845 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-istio-data\") pod \"9ea6c627-8885-490d-88ee-10befd58f673\" (UID: \"9ea6c627-8885-490d-88ee-10befd58f673\") " Oct 30 21:50:02.898961 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.898722 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea6c627-8885-490d-88ee-10befd58f673-istiod-ca-cert" pod "9ea6c627-8885-490d-88ee-10befd58f673" (UID: "9ea6c627-8885-490d-88ee-10befd58f673"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:02.898961 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.898930 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-istio-data" pod "9ea6c627-8885-490d-88ee-10befd58f673" (UID: "9ea6c627-8885-490d-88ee-10befd58f673"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:02.899155 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.899103 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-workload-socket" pod "9ea6c627-8885-490d-88ee-10befd58f673" (UID: "9ea6c627-8885-490d-88ee-10befd58f673"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:02.899490 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.899442 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-workload-certs" pod "9ea6c627-8885-490d-88ee-10befd58f673" (UID: "9ea6c627-8885-490d-88ee-10befd58f673"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:02.899683 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.899658 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-credential-socket" pod "9ea6c627-8885-490d-88ee-10befd58f673" (UID: "9ea6c627-8885-490d-88ee-10befd58f673"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:02.900549 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.900453 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea6c627-8885-490d-88ee-10befd58f673-istio-token" pod "9ea6c627-8885-490d-88ee-10befd58f673" (UID: "9ea6c627-8885-490d-88ee-10befd58f673"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:02.903768 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.903722 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ea6c627-8885-490d-88ee-10befd58f673-istio-podinfo" pod "9ea6c627-8885-490d-88ee-10befd58f673" (UID: "9ea6c627-8885-490d-88ee-10befd58f673"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:02.904977 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.904951 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-istio-envoy" pod "9ea6c627-8885-490d-88ee-10befd58f673" (UID: "9ea6c627-8885-490d-88ee-10befd58f673"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:02.906560 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.906538 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea6c627-8885-490d-88ee-10befd58f673-kube-api-access-8tl2b" pod "9ea6c627-8885-490d-88ee-10befd58f673" (UID: "9ea6c627-8885-490d-88ee-10befd58f673"). InnerVolumeSpecName "kube-api-access-8tl2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:02.999780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.999727 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:02.999780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.999777 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:02.999780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.999787 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ea6c627-8885-490d-88ee-10befd58f673-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.000025 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.999797 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tl2b\" (UniqueName: \"kubernetes.io/projected/9ea6c627-8885-490d-88ee-10befd58f673-kube-api-access-8tl2b\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.000025 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.999807 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.000025 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.999815 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.000025 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.999824 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ea6c627-8885-490d-88ee-10befd58f673-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.000025 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.999832 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ea6c627-8885-490d-88ee-10befd58f673-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.000025 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:02.999839 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ea6c627-8885-490d-88ee-10befd58f673-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.175825 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:03.174233 5801 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 21:50:03.175825 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:03.174306 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" podUID="70abeac9-ada5-4b73-8242-5fb3cad8e5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 21:50:03.184358 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:03.184319 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:03.526410 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:03.526330 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b99efd6a-6f7d-444a-89dd-bf292349a7cc" path="/var/lib/kubelet/pods/b99efd6a-6f7d-444a-89dd-bf292349a7cc/volumes" Oct 30 21:50:03.637128 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:03.637091 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:03.690773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:03.690322 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" event={"ID":"9ea6c627-8885-490d-88ee-10befd58f673","Type":"ContainerDied","Data":"4092d61b2c12339c43ac17e13e617f930f65252c2c403298e3de61064cd77d43"} Oct 30 21:50:03.690773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:03.690403 5801 scope.go:122] "RemoveContainer" containerID="a11e2faeda89f468c8c762c0506a20872a9c71771fa904f821982ddf4cf4501e" Oct 30 21:50:03.690773 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:03.690554 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm" Oct 30 21:50:03.725630 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:03.725594 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm"] Oct 30 21:50:03.730557 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:03.730528 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-000f64-openshift-default-679c88c5b9-4sfrm"] Oct 30 21:50:04.114633 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:04.114592 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:04.178776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:04.175153 5801 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 21:50:04.178776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:04.175214 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" podUID="70abeac9-ada5-4b73-8242-5fb3cad8e5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 21:50:04.213219 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:04.213153 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:04.681058 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:04.680995 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz" podUID="92f3e0a6-905e-46f6-a273-da4ee1e0e3f2" containerName="payload-processing" containerID="cri-o://2a3ad39d38ac25dc5fd24059ff73f064df98a83f48134de290ef47ec1ff2c0d0" gracePeriod=25 Oct 30 21:50:04.952320 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:04.952219 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:05.074116 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:05.074071 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:05.092813 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:05.091333 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz" Oct 30 21:50:05.151872 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:05.151838 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92f3e0a6-905e-46f6-a273-da4ee1e0e3f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f3e0a6-905e-46f6-a273-da4ee1e0e3f2-plugins-config-volume\") pod \"92f3e0a6-905e-46f6-a273-da4ee1e0e3f2\" (UID: \"92f3e0a6-905e-46f6-a273-da4ee1e0e3f2\") " Oct 30 21:50:05.152083 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:05.151993 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f3e0a6-905e-46f6-a273-da4ee1e0e3f2-kube-api-access-69wfz\" (UniqueName: \"kubernetes.io/projected/92f3e0a6-905e-46f6-a273-da4ee1e0e3f2-kube-api-access-69wfz\") pod \"92f3e0a6-905e-46f6-a273-da4ee1e0e3f2\" (UID: \"92f3e0a6-905e-46f6-a273-da4ee1e0e3f2\") " Oct 30 21:50:05.152295 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:05.152269 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f3e0a6-905e-46f6-a273-da4ee1e0e3f2-plugins-config-volume" pod "92f3e0a6-905e-46f6-a273-da4ee1e0e3f2" (UID: "92f3e0a6-905e-46f6-a273-da4ee1e0e3f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:05.176778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:05.152937 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f3e0a6-905e-46f6-a273-da4ee1e0e3f2-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:05.183883 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:05.183843 5801 patch_prober.go:29] interesting pod/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 21:50:05.184318 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:05.183905 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" podUID="70abeac9-ada5-4b73-8242-5fb3cad8e5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 21:50:05.185271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:05.185246 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f3e0a6-905e-46f6-a273-da4ee1e0e3f2-kube-api-access-69wfz" pod "92f3e0a6-905e-46f6-a273-da4ee1e0e3f2" (UID: "92f3e0a6-905e-46f6-a273-da4ee1e0e3f2"). InnerVolumeSpecName "kube-api-access-69wfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:05.277354 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:05.277260 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69wfz\" (UniqueName: \"kubernetes.io/projected/92f3e0a6-905e-46f6-a273-da4ee1e0e3f2-kube-api-access-69wfz\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:05.528858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:05.528744 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ea6c627-8885-490d-88ee-10befd58f673" path="/var/lib/kubelet/pods/9ea6c627-8885-490d-88ee-10befd58f673/volumes" Oct 30 21:50:05.613497 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:05.612969 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6" podUID="8efeccb5-6c34-4aa7-9470-9611f088505f" containerName="payload-pre-processing" containerID="cri-o://fab7978257ff4ecd92529284d01d8d04ab5c9fb0f2a5ab41f28de052e82c586a" gracePeriod=25 Oct 30 21:50:05.713098 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:05.713052 5801 generic.go:356] "Generic (PLEG): container finished" podID="92f3e0a6-905e-46f6-a273-da4ee1e0e3f2" containerID="2a3ad39d38ac25dc5fd24059ff73f064df98a83f48134de290ef47ec1ff2c0d0" exitCode=0 Oct 30 21:50:05.713291 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:05.713105 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz" Oct 30 21:50:05.713291 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:05.713098 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz" event={"ID":"92f3e0a6-905e-46f6-a273-da4ee1e0e3f2","Type":"ContainerDied","Data":"2a3ad39d38ac25dc5fd24059ff73f064df98a83f48134de290ef47ec1ff2c0d0"} Oct 30 21:50:05.713291 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:05.713265 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz" event={"ID":"92f3e0a6-905e-46f6-a273-da4ee1e0e3f2","Type":"ContainerDied","Data":"53fdca7e455d1f0506e831b8cc7c65128944388ab95a4bc0a06bbf8347732b92"} Oct 30 21:50:05.713291 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:05.713292 5801 scope.go:122] "RemoveContainer" containerID="2a3ad39d38ac25dc5fd24059ff73f064df98a83f48134de290ef47ec1ff2c0d0" Oct 30 21:50:05.740484 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:05.740450 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz"] Oct 30 21:50:05.744237 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:05.744210 5801 scope.go:122] "RemoveContainer" containerID="2a3ad39d38ac25dc5fd24059ff73f064df98a83f48134de290ef47ec1ff2c0d0" Oct 30 21:50:05.744635 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:50:05.744611 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a3ad39d38ac25dc5fd24059ff73f064df98a83f48134de290ef47ec1ff2c0d0\": container with ID starting with 2a3ad39d38ac25dc5fd24059ff73f064df98a83f48134de290ef47ec1ff2c0d0 not found: ID does not exist" containerID="2a3ad39d38ac25dc5fd24059ff73f064df98a83f48134de290ef47ec1ff2c0d0" Oct 30 21:50:05.744721 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:05.744647 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a3ad39d38ac25dc5fd24059ff73f064df98a83f48134de290ef47ec1ff2c0d0"} err="failed to get container status \"2a3ad39d38ac25dc5fd24059ff73f064df98a83f48134de290ef47ec1ff2c0d0\": rpc error: code = NotFound desc = could not find container \"2a3ad39d38ac25dc5fd24059ff73f064df98a83f48134de290ef47ec1ff2c0d0\": container with ID starting with 2a3ad39d38ac25dc5fd24059ff73f064df98a83f48134de290ef47ec1ff2c0d0 not found: ID does not exist" Oct 30 21:50:05.747479 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:05.747452 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-950784-7745ddf9f9-b4zzz"] Oct 30 21:50:05.868387 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:05.868349 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:06.001409 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:06.001382 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6" Oct 30 21:50:06.112926 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:06.112896 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8efeccb5-6c34-4aa7-9470-9611f088505f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efeccb5-6c34-4aa7-9470-9611f088505f-plugins-config-volume\") pod \"8efeccb5-6c34-4aa7-9470-9611f088505f\" (UID: \"8efeccb5-6c34-4aa7-9470-9611f088505f\") " Oct 30 21:50:06.113103 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:06.113086 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8efeccb5-6c34-4aa7-9470-9611f088505f-kube-api-access-2nzkh\" (UniqueName: \"kubernetes.io/projected/8efeccb5-6c34-4aa7-9470-9611f088505f-kube-api-access-2nzkh\") pod \"8efeccb5-6c34-4aa7-9470-9611f088505f\" (UID: \"8efeccb5-6c34-4aa7-9470-9611f088505f\") " Oct 30 21:50:06.113280 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:06.113258 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8efeccb5-6c34-4aa7-9470-9611f088505f-plugins-config-volume" pod "8efeccb5-6c34-4aa7-9470-9611f088505f" (UID: "8efeccb5-6c34-4aa7-9470-9611f088505f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:06.113576 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:06.113561 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8efeccb5-6c34-4aa7-9470-9611f088505f-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:06.115180 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:06.115160 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8efeccb5-6c34-4aa7-9470-9611f088505f-kube-api-access-2nzkh" pod "8efeccb5-6c34-4aa7-9470-9611f088505f" (UID: "8efeccb5-6c34-4aa7-9470-9611f088505f"). InnerVolumeSpecName "kube-api-access-2nzkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:06.176513 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:06.176480 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:50:06.176961 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:06.176947 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:50:06.177695 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:06.177676 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:50:06.215404 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:06.215368 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nzkh\" (UniqueName: \"kubernetes.io/projected/8efeccb5-6c34-4aa7-9470-9611f088505f-kube-api-access-2nzkh\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:06.687739 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:06.687699 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:06.725768 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:06.725723 5801 generic.go:356] "Generic (PLEG): container finished" podID="8efeccb5-6c34-4aa7-9470-9611f088505f" containerID="fab7978257ff4ecd92529284d01d8d04ab5c9fb0f2a5ab41f28de052e82c586a" exitCode=0 Oct 30 21:50:06.725948 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:06.725789 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6" Oct 30 21:50:06.725948 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:06.725790 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6" event={"ID":"8efeccb5-6c34-4aa7-9470-9611f088505f","Type":"ContainerDied","Data":"fab7978257ff4ecd92529284d01d8d04ab5c9fb0f2a5ab41f28de052e82c586a"} Oct 30 21:50:06.725948 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:06.725851 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6" event={"ID":"8efeccb5-6c34-4aa7-9470-9611f088505f","Type":"ContainerDied","Data":"305cc7a2fce70ad2684503a3bcb872aa6df6400d7dfd87fcbe9bfcb976ce25f7"} Oct 30 21:50:06.725948 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:06.725880 5801 scope.go:122] "RemoveContainer" containerID="fab7978257ff4ecd92529284d01d8d04ab5c9fb0f2a5ab41f28de052e82c586a" Oct 30 21:50:06.757582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:06.757549 5801 scope.go:122] "RemoveContainer" containerID="fab7978257ff4ecd92529284d01d8d04ab5c9fb0f2a5ab41f28de052e82c586a" Oct 30 21:50:06.758019 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:50:06.757994 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab7978257ff4ecd92529284d01d8d04ab5c9fb0f2a5ab41f28de052e82c586a\": container with ID starting with fab7978257ff4ecd92529284d01d8d04ab5c9fb0f2a5ab41f28de052e82c586a not found: ID does not exist" containerID="fab7978257ff4ecd92529284d01d8d04ab5c9fb0f2a5ab41f28de052e82c586a" Oct 30 21:50:06.758110 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:06.758054 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab7978257ff4ecd92529284d01d8d04ab5c9fb0f2a5ab41f28de052e82c586a"} err="failed to get container status \"fab7978257ff4ecd92529284d01d8d04ab5c9fb0f2a5ab41f28de052e82c586a\": rpc error: code = NotFound desc = could not find container \"fab7978257ff4ecd92529284d01d8d04ab5c9fb0f2a5ab41f28de052e82c586a\": container with ID starting with fab7978257ff4ecd92529284d01d8d04ab5c9fb0f2a5ab41f28de052e82c586a not found: ID does not exist" Oct 30 21:50:06.765549 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:06.765523 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6"] Oct 30 21:50:06.769591 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:06.769567 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-950784-57fb5b6867-s5hg6"] Oct 30 21:50:07.440381 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:07.440343 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:07.526887 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:07.526852 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8efeccb5-6c34-4aa7-9470-9611f088505f" path="/var/lib/kubelet/pods/8efeccb5-6c34-4aa7-9470-9611f088505f/volumes" Oct 30 21:50:07.527425 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:07.527403 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92f3e0a6-905e-46f6-a273-da4ee1e0e3f2" path="/var/lib/kubelet/pods/92f3e0a6-905e-46f6-a273-da4ee1e0e3f2/volumes" Oct 30 21:50:08.131168 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:08.131135 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:08.300552 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:08.300517 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:08.396853 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:08.396768 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:08.632122 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:08.632082 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg"] Oct 30 21:50:08.632644 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:08.632466 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" podUID="8690c196-570d-4604-bfee-2a5d84997f90" containerName="istio-proxy" containerID="cri-o://ac32c88b29e53e1bc4a7b8d4d87a260e8047df0dde8d567a338c6635d17cf8d0" gracePeriod=30 Oct 30 21:50:08.660270 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:08.660162 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:09.190685 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.190402 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:09.307928 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.307894 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp"] Oct 30 21:50:09.308549 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.308535 5801 cpu_manager.go:414] "containerMap: removing container" podUID="8efeccb5-6c34-4aa7-9470-9611f088505f" containerName="payload-pre-processing" Oct 30 21:50:09.308601 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.308550 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8efeccb5-6c34-4aa7-9470-9611f088505f" containerName="payload-pre-processing" Oct 30 21:50:09.308601 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.308565 5801 cpu_manager.go:414] "containerMap: removing container" podUID="9ea6c627-8885-490d-88ee-10befd58f673" containerName="istio-proxy" Oct 30 21:50:09.308601 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.308571 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ea6c627-8885-490d-88ee-10befd58f673" containerName="istio-proxy" Oct 30 21:50:09.308601 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.308586 5801 cpu_manager.go:414] "containerMap: removing container" podUID="92f3e0a6-905e-46f6-a273-da4ee1e0e3f2" containerName="payload-processing" Oct 30 21:50:09.308601 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.308592 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92f3e0a6-905e-46f6-a273-da4ee1e0e3f2" containerName="payload-processing" Oct 30 21:50:09.308785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.308606 5801 cpu_manager.go:414] "containerMap: removing container" podUID="b99efd6a-6f7d-444a-89dd-bf292349a7cc" containerName="istio-proxy" Oct 30 21:50:09.308785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.308611 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b99efd6a-6f7d-444a-89dd-bf292349a7cc" containerName="istio-proxy" Oct 30 21:50:09.308785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.308710 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ea6c627-8885-490d-88ee-10befd58f673" containerName="istio-proxy" Oct 30 21:50:09.308785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.308719 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="b99efd6a-6f7d-444a-89dd-bf292349a7cc" containerName="istio-proxy" Oct 30 21:50:09.308785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.308733 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="8efeccb5-6c34-4aa7-9470-9611f088505f" containerName="payload-pre-processing" Oct 30 21:50:09.308785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.308743 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="92f3e0a6-905e-46f6-a273-da4ee1e0e3f2" containerName="payload-processing" Oct 30 21:50:09.313523 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.313499 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp" Oct 30 21:50:09.321714 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.321692 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:50:09.321872 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.321811 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:50:09.322416 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.322380 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:50:09.322530 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.322507 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:50:09.332248 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.332224 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp"] Oct 30 21:50:09.372368 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.372333 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btwlz\" (UniqueName: \"kubernetes.io/projected/55166a7a-d549-401f-bc28-3f90b8774a59-kube-api-access-btwlz\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp\" (UID: \"55166a7a-d549-401f-bc28-3f90b8774a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp" Oct 30 21:50:09.372538 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.372478 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55166a7a-d549-401f-bc28-3f90b8774a59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp\" (UID: \"55166a7a-d549-401f-bc28-3f90b8774a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp" Oct 30 21:50:09.372617 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.372601 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55166a7a-d549-401f-bc28-3f90b8774a59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp\" (UID: \"55166a7a-d549-401f-bc28-3f90b8774a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp" Oct 30 21:50:09.474253 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.474171 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btwlz\" (UniqueName: \"kubernetes.io/projected/55166a7a-d549-401f-bc28-3f90b8774a59-kube-api-access-btwlz\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp\" (UID: \"55166a7a-d549-401f-bc28-3f90b8774a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp" Oct 30 21:50:09.474393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.474271 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55166a7a-d549-401f-bc28-3f90b8774a59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp\" (UID: \"55166a7a-d549-401f-bc28-3f90b8774a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp" Oct 30 21:50:09.474393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.474372 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55166a7a-d549-401f-bc28-3f90b8774a59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp\" (UID: \"55166a7a-d549-401f-bc28-3f90b8774a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp" Oct 30 21:50:09.474959 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.474935 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55166a7a-d549-401f-bc28-3f90b8774a59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp\" (UID: \"55166a7a-d549-401f-bc28-3f90b8774a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp" Oct 30 21:50:09.476680 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.476656 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55166a7a-d549-401f-bc28-3f90b8774a59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp\" (UID: \"55166a7a-d549-401f-bc28-3f90b8774a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp" Oct 30 21:50:09.503022 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.502988 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btwlz\" (UniqueName: \"kubernetes.io/projected/55166a7a-d549-401f-bc28-3f90b8774a59-kube-api-access-btwlz\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp\" (UID: \"55166a7a-d549-401f-bc28-3f90b8774a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp" Oct 30 21:50:09.628394 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.628361 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp" Oct 30 21:50:09.925659 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:50:09.925620 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55166a7a_d549_401f_bc28_3f90b8774a59.slice/crio-336a5032391e3314767f19cf3aaeea1eaf8f974fdb365f7d9fbd0b242198c71e WatchSource:0}: Error finding container 336a5032391e3314767f19cf3aaeea1eaf8f974fdb365f7d9fbd0b242198c71e: Status 404 returned error can't find the container with id 336a5032391e3314767f19cf3aaeea1eaf8f974fdb365f7d9fbd0b242198c71e Oct 30 21:50:09.926108 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:09.926011 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp"] Oct 30 21:50:10.336476 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:10.336446 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:10.863654 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:10.863612 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp" event={"ID":"55166a7a-d549-401f-bc28-3f90b8774a59","Type":"ContainerStarted","Data":"3888199eff371d1575e24c5168dd6330f1bad78de23f04a6cf5fc74e9f6f91d8"} Oct 30 21:50:10.863879 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:10.863682 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp" event={"ID":"55166a7a-d549-401f-bc28-3f90b8774a59","Type":"ContainerStarted","Data":"336a5032391e3314767f19cf3aaeea1eaf8f974fdb365f7d9fbd0b242198c71e"} Oct 30 21:50:10.888382 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:10.888329 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp" podStartSLOduration=1.888311367 podStartE2EDuration="1.888311367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:10.881426226 +0000 UTC m=+1279.477242889" watchObservedRunningTime="2026-10-30 21:50:10.888311367 +0000 UTC m=+1279.484128029" Oct 30 21:50:11.287622 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:11.287537 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:11.691522 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:11.691489 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:11.791736 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:11.791699 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:11.877143 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:11.874117 5801 generic.go:356] "Generic (PLEG): container finished" podID="55166a7a-d549-401f-bc28-3f90b8774a59" containerID="3888199eff371d1575e24c5168dd6330f1bad78de23f04a6cf5fc74e9f6f91d8" exitCode=0 Oct 30 21:50:11.877143 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:11.874199 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp" event={"ID":"55166a7a-d549-401f-bc28-3f90b8774a59","Type":"ContainerDied","Data":"3888199eff371d1575e24c5168dd6330f1bad78de23f04a6cf5fc74e9f6f91d8"} Oct 30 21:50:12.033002 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:12.032923 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt"] Oct 30 21:50:12.033190 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:12.033164 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" podUID="a26eafe2-a22a-47a0-ae23-562931b2cead" containerName="istio-proxy" containerID="cri-o://84614df4a9236c9a2c63fae4cf0542c00e1e9b6397ceb861b2a3da72f0b70333" gracePeriod=30 Oct 30 21:50:12.040937 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:12.040908 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt"] Oct 30 21:50:12.041109 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:12.041086 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" podUID="409f33be-0503-4f36-ad79-93d1646f51ec" containerName="istio-proxy" containerID="cri-o://0b29388fe91c2329a24c37d18faade709132c7a3946366de57909f4659b79126" gracePeriod=30 Oct 30 21:50:12.047512 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:12.047485 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v"] Oct 30 21:50:12.047970 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:12.047936 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" podUID="70abeac9-ada5-4b73-8242-5fb3cad8e5a6" containerName="istio-proxy" containerID="cri-o://8e69e57bf3f8c4647ad767fddd664826b57b23600102c0cd5c9a787f512f920a" gracePeriod=30 Oct 30 21:50:12.075740 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:12.075715 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:12.269672 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:12.268065 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:13.043141 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.043115 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp" Oct 30 21:50:13.096130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.096082 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:13.160235 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.160140 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55166a7a-d549-401f-bc28-3f90b8774a59-kube-api-access-btwlz\" (UniqueName: \"kubernetes.io/projected/55166a7a-d549-401f-bc28-3f90b8774a59-kube-api-access-btwlz\") pod \"55166a7a-d549-401f-bc28-3f90b8774a59\" (UID: \"55166a7a-d549-401f-bc28-3f90b8774a59\") " Oct 30 21:50:13.160491 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.160466 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55166a7a-d549-401f-bc28-3f90b8774a59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55166a7a-d549-401f-bc28-3f90b8774a59-maas-api-openshift-ca\") pod \"55166a7a-d549-401f-bc28-3f90b8774a59\" (UID: \"55166a7a-d549-401f-bc28-3f90b8774a59\") " Oct 30 21:50:13.160587 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.160520 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55166a7a-d549-401f-bc28-3f90b8774a59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55166a7a-d549-401f-bc28-3f90b8774a59-maas-api-xks-ca\") pod \"55166a7a-d549-401f-bc28-3f90b8774a59\" (UID: \"55166a7a-d549-401f-bc28-3f90b8774a59\") " Oct 30 21:50:13.161730 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.161696 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55166a7a-d549-401f-bc28-3f90b8774a59-maas-api-openshift-ca" pod "55166a7a-d549-401f-bc28-3f90b8774a59" (UID: "55166a7a-d549-401f-bc28-3f90b8774a59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:13.162545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.162501 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55166a7a-d549-401f-bc28-3f90b8774a59-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.164254 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.164227 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55166a7a-d549-401f-bc28-3f90b8774a59-kube-api-access-btwlz" pod "55166a7a-d549-401f-bc28-3f90b8774a59" (UID: "55166a7a-d549-401f-bc28-3f90b8774a59"). InnerVolumeSpecName "kube-api-access-btwlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:13.165883 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.165858 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55166a7a-d549-401f-bc28-3f90b8774a59-maas-api-xks-ca" pod "55166a7a-d549-401f-bc28-3f90b8774a59" (UID: "55166a7a-d549-401f-bc28-3f90b8774a59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:13.264030 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.263996 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55166a7a-d549-401f-bc28-3f90b8774a59-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.264030 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.264027 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btwlz\" (UniqueName: \"kubernetes.io/projected/55166a7a-d549-401f-bc28-3f90b8774a59-kube-api-access-btwlz\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.510701 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.510618 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58c5d82-cm72n"] Oct 30 21:50:13.511295 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.511277 5801 cpu_manager.go:414] "containerMap: removing container" podUID="9ea6c627-8885-490d-88ee-10befd58f673" containerName="istio-proxy" Oct 30 21:50:13.511295 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.511295 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ea6c627-8885-490d-88ee-10befd58f673" containerName="istio-proxy" Oct 30 21:50:13.511417 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.511332 5801 cpu_manager.go:414] "containerMap: removing container" podUID="55166a7a-d549-401f-bc28-3f90b8774a59" containerName="revoke-keys" Oct 30 21:50:13.511417 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.511339 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55166a7a-d549-401f-bc28-3f90b8774a59" containerName="revoke-keys" Oct 30 21:50:13.511532 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.511515 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ea6c627-8885-490d-88ee-10befd58f673" containerName="istio-proxy" Oct 30 21:50:13.511592 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.511541 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="55166a7a-d549-401f-bc28-3f90b8774a59" containerName="revoke-keys" Oct 30 21:50:13.515196 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.515179 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58c5d82-cm72n" Oct 30 21:50:13.535284 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.531307 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58c5d82-cm72n"] Oct 30 21:50:13.568031 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.568005 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce20b749-0d51-4e79-a243-109b758e70f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f58c5d82-cm72n\" (UID: \"ce20b749-0d51-4e79-a243-109b758e70f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58c5d82-cm72n" Oct 30 21:50:13.568140 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.568062 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce20b749-0d51-4e79-a243-109b758e70f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f58c5d82-cm72n\" (UID: \"ce20b749-0d51-4e79-a243-109b758e70f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58c5d82-cm72n" Oct 30 21:50:13.568206 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.568188 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvj4k\" (UniqueName: \"kubernetes.io/projected/ce20b749-0d51-4e79-a243-109b758e70f8-kube-api-access-kvj4k\") pod \"maas-api-revoke-sub-f58c5d82-cm72n\" (UID: \"ce20b749-0d51-4e79-a243-109b758e70f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58c5d82-cm72n" Oct 30 21:50:13.670173 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.670149 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvj4k\" (UniqueName: \"kubernetes.io/projected/ce20b749-0d51-4e79-a243-109b758e70f8-kube-api-access-kvj4k\") pod \"maas-api-revoke-sub-f58c5d82-cm72n\" (UID: \"ce20b749-0d51-4e79-a243-109b758e70f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58c5d82-cm72n" Oct 30 21:50:13.670340 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.670324 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce20b749-0d51-4e79-a243-109b758e70f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f58c5d82-cm72n\" (UID: \"ce20b749-0d51-4e79-a243-109b758e70f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58c5d82-cm72n" Oct 30 21:50:13.670404 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.670363 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce20b749-0d51-4e79-a243-109b758e70f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f58c5d82-cm72n\" (UID: \"ce20b749-0d51-4e79-a243-109b758e70f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58c5d82-cm72n" Oct 30 21:50:13.670954 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.670934 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce20b749-0d51-4e79-a243-109b758e70f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f58c5d82-cm72n\" (UID: \"ce20b749-0d51-4e79-a243-109b758e70f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58c5d82-cm72n" Oct 30 21:50:13.672546 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.672527 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce20b749-0d51-4e79-a243-109b758e70f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f58c5d82-cm72n\" (UID: \"ce20b749-0d51-4e79-a243-109b758e70f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58c5d82-cm72n" Oct 30 21:50:13.681498 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.681478 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvj4k\" (UniqueName: \"kubernetes.io/projected/ce20b749-0d51-4e79-a243-109b758e70f8-kube-api-access-kvj4k\") pod \"maas-api-revoke-sub-f58c5d82-cm72n\" (UID: \"ce20b749-0d51-4e79-a243-109b758e70f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58c5d82-cm72n" Oct 30 21:50:13.828657 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.828598 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:13.837259 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.837230 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58c5d82-cm72n" Oct 30 21:50:13.912333 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.912296 5801 generic.go:356] "Generic (PLEG): container finished" podID="8690c196-570d-4604-bfee-2a5d84997f90" containerID="ac32c88b29e53e1bc4a7b8d4d87a260e8047df0dde8d567a338c6635d17cf8d0" exitCode=0 Oct 30 21:50:13.912818 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.912740 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" event={"ID":"8690c196-570d-4604-bfee-2a5d84997f90","Type":"ContainerDied","Data":"ac32c88b29e53e1bc4a7b8d4d87a260e8047df0dde8d567a338c6635d17cf8d0"} Oct 30 21:50:13.913066 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.913041 5801 scope.go:122] "RemoveContainer" containerID="a2ee38c889b76a672392ba2971ab7582ffd557d721883da3214c4da9555829ce" Oct 30 21:50:13.930922 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.930889 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp" event={"ID":"55166a7a-d549-401f-bc28-3f90b8774a59","Type":"ContainerDied","Data":"336a5032391e3314767f19cf3aaeea1eaf8f974fdb365f7d9fbd0b242198c71e"} Oct 30 21:50:13.931428 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.931333 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="336a5032391e3314767f19cf3aaeea1eaf8f974fdb365f7d9fbd0b242198c71e" Oct 30 21:50:13.932321 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.931600 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp" Oct 30 21:50:13.947656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:13.947630 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:50:14.099179 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.099100 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-istio-data\") pod \"8690c196-570d-4604-bfee-2a5d84997f90\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " Oct 30 21:50:14.099179 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.099156 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8690c196-570d-4604-bfee-2a5d84997f90-kube-api-access-wxmtp\" (UniqueName: \"kubernetes.io/projected/8690c196-570d-4604-bfee-2a5d84997f90-kube-api-access-wxmtp\") pod \"8690c196-570d-4604-bfee-2a5d84997f90\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " Oct 30 21:50:14.099656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.099255 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8690c196-570d-4604-bfee-2a5d84997f90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8690c196-570d-4604-bfee-2a5d84997f90-istio-podinfo\") pod \"8690c196-570d-4604-bfee-2a5d84997f90\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " Oct 30 21:50:14.099656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.099277 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-credential-socket\") pod \"8690c196-570d-4604-bfee-2a5d84997f90\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " Oct 30 21:50:14.099656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.099319 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8690c196-570d-4604-bfee-2a5d84997f90-istio-token\" (UniqueName: \"kubernetes.io/projected/8690c196-570d-4604-bfee-2a5d84997f90-istio-token\") pod \"8690c196-570d-4604-bfee-2a5d84997f90\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " Oct 30 21:50:14.099656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.099390 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-istio-envoy\") pod \"8690c196-570d-4604-bfee-2a5d84997f90\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " Oct 30 21:50:14.099656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.099466 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8690c196-570d-4604-bfee-2a5d84997f90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8690c196-570d-4604-bfee-2a5d84997f90-istiod-ca-cert\") pod \"8690c196-570d-4604-bfee-2a5d84997f90\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " Oct 30 21:50:14.099656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.099505 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-workload-certs\") pod \"8690c196-570d-4604-bfee-2a5d84997f90\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " Oct 30 21:50:14.099656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.099537 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-istio-data" pod "8690c196-570d-4604-bfee-2a5d84997f90" (UID: "8690c196-570d-4604-bfee-2a5d84997f90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:14.099656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.099553 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-credential-socket" pod "8690c196-570d-4604-bfee-2a5d84997f90" (UID: "8690c196-570d-4604-bfee-2a5d84997f90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:14.099656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.099551 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-workload-socket\") pod \"8690c196-570d-4604-bfee-2a5d84997f90\" (UID: \"8690c196-570d-4604-bfee-2a5d84997f90\") " Oct 30 21:50:14.100214 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.099806 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-workload-socket" pod "8690c196-570d-4604-bfee-2a5d84997f90" (UID: "8690c196-570d-4604-bfee-2a5d84997f90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:14.100214 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.099833 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-workload-certs" pod "8690c196-570d-4604-bfee-2a5d84997f90" (UID: "8690c196-570d-4604-bfee-2a5d84997f90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:14.100214 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.099921 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8690c196-570d-4604-bfee-2a5d84997f90-istiod-ca-cert" pod "8690c196-570d-4604-bfee-2a5d84997f90" (UID: "8690c196-570d-4604-bfee-2a5d84997f90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:14.101724 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.101697 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-istio-envoy" pod "8690c196-570d-4604-bfee-2a5d84997f90" (UID: "8690c196-570d-4604-bfee-2a5d84997f90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:14.101724 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.101708 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8690c196-570d-4604-bfee-2a5d84997f90-istio-token" pod "8690c196-570d-4604-bfee-2a5d84997f90" (UID: "8690c196-570d-4604-bfee-2a5d84997f90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:14.102019 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.101727 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8690c196-570d-4604-bfee-2a5d84997f90-kube-api-access-wxmtp" pod "8690c196-570d-4604-bfee-2a5d84997f90" (UID: "8690c196-570d-4604-bfee-2a5d84997f90"). InnerVolumeSpecName "kube-api-access-wxmtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:14.102019 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.101809 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8690c196-570d-4604-bfee-2a5d84997f90-istio-podinfo" pod "8690c196-570d-4604-bfee-2a5d84997f90" (UID: "8690c196-570d-4604-bfee-2a5d84997f90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:14.102138 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.102084 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8690c196-570d-4604-bfee-2a5d84997f90-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:14.102138 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.102107 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:14.102138 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.102121 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:14.102138 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.102135 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:14.102297 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.102149 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxmtp\" (UniqueName: \"kubernetes.io/projected/8690c196-570d-4604-bfee-2a5d84997f90-kube-api-access-wxmtp\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:14.102297 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.102165 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8690c196-570d-4604-bfee-2a5d84997f90-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:14.102297 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.102178 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:14.102297 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.102191 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8690c196-570d-4604-bfee-2a5d84997f90-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:14.102297 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.102204 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8690c196-570d-4604-bfee-2a5d84997f90-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:14.162369 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:50:14.162340 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce20b749_0d51_4e79_a243_109b758e70f8.slice/crio-aec995878667f36acdcd4041206440f664c1e27d640e26c5ffb5942d2da7ec94 WatchSource:0}: Error finding container aec995878667f36acdcd4041206440f664c1e27d640e26c5ffb5942d2da7ec94: Status 404 returned error can't find the container with id aec995878667f36acdcd4041206440f664c1e27d640e26c5ffb5942d2da7ec94 Oct 30 21:50:14.162621 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.162599 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58c5d82-cm72n"] Oct 30 21:50:14.655498 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.655420 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:14.941173 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.941076 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58c5d82-cm72n" event={"ID":"ce20b749-0d51-4e79-a243-109b758e70f8","Type":"ContainerStarted","Data":"ac1cdae13c15fa97882230b8062f5efbc74f6e9dc26b01a47719a4339de103ce"} Oct 30 21:50:14.941173 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.941144 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58c5d82-cm72n" event={"ID":"ce20b749-0d51-4e79-a243-109b758e70f8","Type":"ContainerStarted","Data":"aec995878667f36acdcd4041206440f664c1e27d640e26c5ffb5942d2da7ec94"} Oct 30 21:50:14.942968 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.942946 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" Oct 30 21:50:14.943107 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.942950 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg" event={"ID":"8690c196-570d-4604-bfee-2a5d84997f90","Type":"ContainerDied","Data":"a321b25b4f45c29101d29f2601ca34ded14c19f9fc0710229bdb24befb3e5af2"} Oct 30 21:50:14.943107 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:14.943062 5801 scope.go:122] "RemoveContainer" containerID="ac32c88b29e53e1bc4a7b8d4d87a260e8047df0dde8d567a338c6635d17cf8d0" Oct 30 21:50:15.003367 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:15.003312 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58c5d82-cm72n" podStartSLOduration=2.003299207 podStartE2EDuration="2.003299207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:15.000851278 +0000 UTC m=+1283.596667939" watchObservedRunningTime="2026-10-30 21:50:15.003299207 +0000 UTC m=+1283.599115866" Oct 30 21:50:15.046367 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:15.046335 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg"] Oct 30 21:50:15.050362 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:15.050339 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2ba0bd-openshift-default-6dc4677897-4wrgg"] Oct 30 21:50:15.525580 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:15.525547 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8690c196-570d-4604-bfee-2a5d84997f90" path="/var/lib/kubelet/pods/8690c196-570d-4604-bfee-2a5d84997f90/volumes" Oct 30 21:50:15.539354 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:15.539325 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:15.644170 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:15.644127 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:15.952770 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:15.952721 5801 generic.go:356] "Generic (PLEG): container finished" podID="ce20b749-0d51-4e79-a243-109b758e70f8" containerID="ac1cdae13c15fa97882230b8062f5efbc74f6e9dc26b01a47719a4339de103ce" exitCode=0 Oct 30 21:50:15.952952 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:15.952779 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58c5d82-cm72n" event={"ID":"ce20b749-0d51-4e79-a243-109b758e70f8","Type":"ContainerDied","Data":"ac1cdae13c15fa97882230b8062f5efbc74f6e9dc26b01a47719a4339de103ce"} Oct 30 21:50:16.358447 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:16.358407 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:16.375801 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:16.375768 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz"] Oct 30 21:50:16.376480 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:16.376465 5801 cpu_manager.go:414] "containerMap: removing container" podUID="8690c196-570d-4604-bfee-2a5d84997f90" containerName="istio-proxy" Oct 30 21:50:16.376523 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:16.376482 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8690c196-570d-4604-bfee-2a5d84997f90" containerName="istio-proxy" Oct 30 21:50:16.376523 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:16.376493 5801 cpu_manager.go:414] "containerMap: removing container" podUID="8690c196-570d-4604-bfee-2a5d84997f90" containerName="istio-proxy" Oct 30 21:50:16.376523 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:16.376499 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8690c196-570d-4604-bfee-2a5d84997f90" containerName="istio-proxy" Oct 30 21:50:16.376617 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:16.376586 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="8690c196-570d-4604-bfee-2a5d84997f90" containerName="istio-proxy" Oct 30 21:50:16.376617 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:16.376595 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="8690c196-570d-4604-bfee-2a5d84997f90" containerName="istio-proxy" Oct 30 21:50:16.380357 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:16.380337 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz" Oct 30 21:50:16.389502 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:16.389479 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz"] Oct 30 21:50:16.543901 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:16.543867 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdx5p\" (UniqueName: \"kubernetes.io/projected/2f2c9cba-2588-4e81-9a7c-33fef0ffb58e-kube-api-access-kdx5p\") pod \"maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz\" (UID: \"2f2c9cba-2588-4e81-9a7c-33fef0ffb58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz" Oct 30 21:50:16.544351 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:16.544104 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2c9cba-2588-4e81-9a7c-33fef0ffb58e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz\" (UID: \"2f2c9cba-2588-4e81-9a7c-33fef0ffb58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz" Oct 30 21:50:16.544351 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:16.544172 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2c9cba-2588-4e81-9a7c-33fef0ffb58e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz\" (UID: \"2f2c9cba-2588-4e81-9a7c-33fef0ffb58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz" Oct 30 21:50:16.645535 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:16.645441 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2c9cba-2588-4e81-9a7c-33fef0ffb58e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz\" (UID: \"2f2c9cba-2588-4e81-9a7c-33fef0ffb58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz" Oct 30 21:50:16.645535 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:16.645519 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdx5p\" (UniqueName: \"kubernetes.io/projected/2f2c9cba-2588-4e81-9a7c-33fef0ffb58e-kube-api-access-kdx5p\") pod \"maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz\" (UID: \"2f2c9cba-2588-4e81-9a7c-33fef0ffb58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz" Oct 30 21:50:16.646035 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:16.646004 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2c9cba-2588-4e81-9a7c-33fef0ffb58e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz\" (UID: \"2f2c9cba-2588-4e81-9a7c-33fef0ffb58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz" Oct 30 21:50:16.646823 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:16.646798 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2c9cba-2588-4e81-9a7c-33fef0ffb58e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz\" (UID: \"2f2c9cba-2588-4e81-9a7c-33fef0ffb58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz" Oct 30 21:50:16.648234 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:16.648216 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2c9cba-2588-4e81-9a7c-33fef0ffb58e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz\" (UID: \"2f2c9cba-2588-4e81-9a7c-33fef0ffb58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz" Oct 30 21:50:16.659385 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:16.658326 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdx5p\" (UniqueName: \"kubernetes.io/projected/2f2c9cba-2588-4e81-9a7c-33fef0ffb58e-kube-api-access-kdx5p\") pod \"maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz\" (UID: \"2f2c9cba-2588-4e81-9a7c-33fef0ffb58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz" Oct 30 21:50:16.697082 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:16.697049 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz" Oct 30 21:50:16.984323 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:50:16.984288 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2c9cba_2588_4e81_9a7c_33fef0ffb58e.slice/crio-87d366d9d3250831bd659147e9f6bf5bff0c4bc028899c5089e24e7d6d907fed WatchSource:0}: Error finding container 87d366d9d3250831bd659147e9f6bf5bff0c4bc028899c5089e24e7d6d907fed: Status 404 returned error can't find the container with id 87d366d9d3250831bd659147e9f6bf5bff0c4bc028899c5089e24e7d6d907fed Oct 30 21:50:16.985839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:16.985810 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz"] Oct 30 21:50:17.194502 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.194473 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58c5d82-cm72n" Oct 30 21:50:17.242041 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.241910 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:17.357851 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.357650 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce20b749-0d51-4e79-a243-109b758e70f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce20b749-0d51-4e79-a243-109b758e70f8-maas-api-openshift-ca\") pod \"ce20b749-0d51-4e79-a243-109b758e70f8\" (UID: \"ce20b749-0d51-4e79-a243-109b758e70f8\") " Oct 30 21:50:17.358986 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.358100 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce20b749-0d51-4e79-a243-109b758e70f8-kube-api-access-kvj4k\" (UniqueName: \"kubernetes.io/projected/ce20b749-0d51-4e79-a243-109b758e70f8-kube-api-access-kvj4k\") pod \"ce20b749-0d51-4e79-a243-109b758e70f8\" (UID: \"ce20b749-0d51-4e79-a243-109b758e70f8\") " Oct 30 21:50:17.358986 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.358216 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce20b749-0d51-4e79-a243-109b758e70f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce20b749-0d51-4e79-a243-109b758e70f8-maas-api-xks-ca\") pod \"ce20b749-0d51-4e79-a243-109b758e70f8\" (UID: \"ce20b749-0d51-4e79-a243-109b758e70f8\") " Oct 30 21:50:17.361859 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.361691 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce20b749-0d51-4e79-a243-109b758e70f8-maas-api-openshift-ca" pod "ce20b749-0d51-4e79-a243-109b758e70f8" (UID: "ce20b749-0d51-4e79-a243-109b758e70f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:17.457540 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.457146 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce20b749-0d51-4e79-a243-109b758e70f8-kube-api-access-kvj4k" pod "ce20b749-0d51-4e79-a243-109b758e70f8" (UID: "ce20b749-0d51-4e79-a243-109b758e70f8"). InnerVolumeSpecName "kube-api-access-kvj4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:17.461052 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.457833 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce20b749-0d51-4e79-a243-109b758e70f8-maas-api-xks-ca" pod "ce20b749-0d51-4e79-a243-109b758e70f8" (UID: "ce20b749-0d51-4e79-a243-109b758e70f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:17.465521 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.465455 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce20b749-0d51-4e79-a243-109b758e70f8-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.465521 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.465483 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvj4k\" (UniqueName: \"kubernetes.io/projected/ce20b749-0d51-4e79-a243-109b758e70f8-kube-api-access-kvj4k\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.465521 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.465500 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce20b749-0d51-4e79-a243-109b758e70f8-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.480256 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.479352 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:50:17.565026 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.564987 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:50:17.567932 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.567909 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-credential-socket\") pod \"a26eafe2-a22a-47a0-ae23-562931b2cead\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " Oct 30 21:50:17.568176 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.568156 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-credential-socket" pod "a26eafe2-a22a-47a0-ae23-562931b2cead" (UID: "a26eafe2-a22a-47a0-ae23-562931b2cead"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.568300 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.568185 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-workload-socket\") pod \"a26eafe2-a22a-47a0-ae23-562931b2cead\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " Oct 30 21:50:17.568394 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.568374 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a26eafe2-a22a-47a0-ae23-562931b2cead-kube-api-access-7kzrd\" (UniqueName: \"kubernetes.io/projected/a26eafe2-a22a-47a0-ae23-562931b2cead-kube-api-access-7kzrd\") pod \"a26eafe2-a22a-47a0-ae23-562931b2cead\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " Oct 30 21:50:17.568486 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.568465 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-workload-socket" pod "a26eafe2-a22a-47a0-ae23-562931b2cead" (UID: "a26eafe2-a22a-47a0-ae23-562931b2cead"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.568549 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.568490 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-podinfo\") pod \"a26eafe2-a22a-47a0-ae23-562931b2cead\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " Oct 30 21:50:17.568606 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.568586 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a26eafe2-a22a-47a0-ae23-562931b2cead-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26eafe2-a22a-47a0-ae23-562931b2cead-istiod-ca-cert\") pod \"a26eafe2-a22a-47a0-ae23-562931b2cead\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " Oct 30 21:50:17.568663 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.568633 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-envoy\") pod \"a26eafe2-a22a-47a0-ae23-562931b2cead\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " Oct 30 21:50:17.568723 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.568705 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-workload-certs\") pod \"a26eafe2-a22a-47a0-ae23-562931b2cead\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " Oct 30 21:50:17.568821 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.568741 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-data\") pod \"a26eafe2-a22a-47a0-ae23-562931b2cead\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " Oct 30 21:50:17.568821 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.568793 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-token\" (UniqueName: \"kubernetes.io/projected/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-token\") pod \"a26eafe2-a22a-47a0-ae23-562931b2cead\" (UID: \"a26eafe2-a22a-47a0-ae23-562931b2cead\") " Oct 30 21:50:17.569517 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.569494 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.569651 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.569634 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.571221 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.571195 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a26eafe2-a22a-47a0-ae23-562931b2cead-kube-api-access-7kzrd" pod "a26eafe2-a22a-47a0-ae23-562931b2cead" (UID: "a26eafe2-a22a-47a0-ae23-562931b2cead"). InnerVolumeSpecName "kube-api-access-7kzrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:17.571314 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.571236 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-workload-certs" pod "a26eafe2-a22a-47a0-ae23-562931b2cead" (UID: "a26eafe2-a22a-47a0-ae23-562931b2cead"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.571561 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.571541 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a26eafe2-a22a-47a0-ae23-562931b2cead-istiod-ca-cert" pod "a26eafe2-a22a-47a0-ae23-562931b2cead" (UID: "a26eafe2-a22a-47a0-ae23-562931b2cead"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:17.571660 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.571500 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-data" pod "a26eafe2-a22a-47a0-ae23-562931b2cead" (UID: "a26eafe2-a22a-47a0-ae23-562931b2cead"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.572487 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.572464 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-envoy" pod "a26eafe2-a22a-47a0-ae23-562931b2cead" (UID: "a26eafe2-a22a-47a0-ae23-562931b2cead"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.572707 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.572690 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:50:17.573406 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.573377 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-podinfo" pod "a26eafe2-a22a-47a0-ae23-562931b2cead" (UID: "a26eafe2-a22a-47a0-ae23-562931b2cead"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:17.573511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.573492 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-token" pod "a26eafe2-a22a-47a0-ae23-562931b2cead" (UID: "a26eafe2-a22a-47a0-ae23-562931b2cead"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:17.670645 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.670612 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-podinfo\") pod \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " Oct 30 21:50:17.670818 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.670660 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-workload-socket\") pod \"409f33be-0503-4f36-ad79-93d1646f51ec\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " Oct 30 21:50:17.670818 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.670787 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/409f33be-0503-4f36-ad79-93d1646f51ec-kube-api-access-7t25v\" (UniqueName: \"kubernetes.io/projected/409f33be-0503-4f36-ad79-93d1646f51ec-kube-api-access-7t25v\") pod \"409f33be-0503-4f36-ad79-93d1646f51ec\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " Oct 30 21:50:17.670954 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.670856 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-workload-certs\") pod \"409f33be-0503-4f36-ad79-93d1646f51ec\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " Oct 30 21:50:17.670954 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.670880 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-workload-socket\") pod \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " Oct 30 21:50:17.670954 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.670942 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-workload-certs\") pod \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " Oct 30 21:50:17.671135 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.670999 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-envoy\") pod \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " Oct 30 21:50:17.671135 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.671047 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/409f33be-0503-4f36-ad79-93d1646f51ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409f33be-0503-4f36-ad79-93d1646f51ec-istiod-ca-cert\") pod \"409f33be-0503-4f36-ad79-93d1646f51ec\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " Oct 30 21:50:17.671135 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.671070 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-kube-api-access-5gg7r\" (UniqueName: \"kubernetes.io/projected/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-kube-api-access-5gg7r\") pod \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " Oct 30 21:50:17.671296 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.671140 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/409f33be-0503-4f36-ad79-93d1646f51ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409f33be-0503-4f36-ad79-93d1646f51ec-istio-podinfo\") pod \"409f33be-0503-4f36-ad79-93d1646f51ec\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " Oct 30 21:50:17.671296 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.671166 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-data\") pod \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " Oct 30 21:50:17.671296 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.671221 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-token\" (UniqueName: \"kubernetes.io/projected/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-token\") pod \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " Oct 30 21:50:17.671460 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.671303 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istiod-ca-cert\") pod \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " Oct 30 21:50:17.671460 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.671337 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/409f33be-0503-4f36-ad79-93d1646f51ec-istio-token\" (UniqueName: \"kubernetes.io/projected/409f33be-0503-4f36-ad79-93d1646f51ec-istio-token\") pod \"409f33be-0503-4f36-ad79-93d1646f51ec\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " Oct 30 21:50:17.671460 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.671413 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-istio-envoy\") pod \"409f33be-0503-4f36-ad79-93d1646f51ec\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " Oct 30 21:50:17.671619 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.671469 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-credential-socket\") pod \"409f33be-0503-4f36-ad79-93d1646f51ec\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " Oct 30 21:50:17.671619 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.671497 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-istio-data\") pod \"409f33be-0503-4f36-ad79-93d1646f51ec\" (UID: \"409f33be-0503-4f36-ad79-93d1646f51ec\") " Oct 30 21:50:17.671619 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.671536 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-credential-socket\") pod \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\" (UID: \"70abeac9-ada5-4b73-8242-5fb3cad8e5a6\") " Oct 30 21:50:17.672279 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.672254 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26eafe2-a22a-47a0-ae23-562931b2cead-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.672394 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.672284 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.672394 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.672297 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.672394 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.672309 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.672394 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.672322 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.672394 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.672335 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kzrd\" (UniqueName: \"kubernetes.io/projected/a26eafe2-a22a-47a0-ae23-562931b2cead-kube-api-access-7kzrd\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.672394 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.672349 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26eafe2-a22a-47a0-ae23-562931b2cead-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.673466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.673440 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-workload-socket" pod "70abeac9-ada5-4b73-8242-5fb3cad8e5a6" (UID: "70abeac9-ada5-4b73-8242-5fb3cad8e5a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.673563 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.673436 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-workload-socket" pod "409f33be-0503-4f36-ad79-93d1646f51ec" (UID: "409f33be-0503-4f36-ad79-93d1646f51ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.673623 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.673605 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-workload-certs" pod "409f33be-0503-4f36-ad79-93d1646f51ec" (UID: "409f33be-0503-4f36-ad79-93d1646f51ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.673783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.673735 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-istio-data" pod "409f33be-0503-4f36-ad79-93d1646f51ec" (UID: "409f33be-0503-4f36-ad79-93d1646f51ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.673977 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.673960 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-credential-socket" pod "409f33be-0503-4f36-ad79-93d1646f51ec" (UID: "409f33be-0503-4f36-ad79-93d1646f51ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.674148 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.674131 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-credential-socket" pod "70abeac9-ada5-4b73-8242-5fb3cad8e5a6" (UID: "70abeac9-ada5-4b73-8242-5fb3cad8e5a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.675269 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.675248 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istiod-ca-cert" pod "70abeac9-ada5-4b73-8242-5fb3cad8e5a6" (UID: "70abeac9-ada5-4b73-8242-5fb3cad8e5a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:17.675478 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.675445 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-workload-certs" pod "70abeac9-ada5-4b73-8242-5fb3cad8e5a6" (UID: "70abeac9-ada5-4b73-8242-5fb3cad8e5a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.675695 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.675679 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/409f33be-0503-4f36-ad79-93d1646f51ec-istiod-ca-cert" pod "409f33be-0503-4f36-ad79-93d1646f51ec" (UID: "409f33be-0503-4f36-ad79-93d1646f51ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:17.675898 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.675882 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-data" pod "70abeac9-ada5-4b73-8242-5fb3cad8e5a6" (UID: "70abeac9-ada5-4b73-8242-5fb3cad8e5a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.676868 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.676832 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-podinfo" pod "70abeac9-ada5-4b73-8242-5fb3cad8e5a6" (UID: "70abeac9-ada5-4b73-8242-5fb3cad8e5a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:17.677508 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.677461 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409f33be-0503-4f36-ad79-93d1646f51ec-kube-api-access-7t25v" pod "409f33be-0503-4f36-ad79-93d1646f51ec" (UID: "409f33be-0503-4f36-ad79-93d1646f51ec"). InnerVolumeSpecName "kube-api-access-7t25v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:17.677706 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.677678 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-istio-envoy" pod "409f33be-0503-4f36-ad79-93d1646f51ec" (UID: "409f33be-0503-4f36-ad79-93d1646f51ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.677964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.677940 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-envoy" pod "70abeac9-ada5-4b73-8242-5fb3cad8e5a6" (UID: "70abeac9-ada5-4b73-8242-5fb3cad8e5a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:17.678416 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.678388 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409f33be-0503-4f36-ad79-93d1646f51ec-istio-token" pod "409f33be-0503-4f36-ad79-93d1646f51ec" (UID: "409f33be-0503-4f36-ad79-93d1646f51ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:17.678984 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.678943 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/409f33be-0503-4f36-ad79-93d1646f51ec-istio-podinfo" pod "409f33be-0503-4f36-ad79-93d1646f51ec" (UID: "409f33be-0503-4f36-ad79-93d1646f51ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:17.679069 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.679032 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-kube-api-access-5gg7r" pod "70abeac9-ada5-4b73-8242-5fb3cad8e5a6" (UID: "70abeac9-ada5-4b73-8242-5fb3cad8e5a6"). InnerVolumeSpecName "kube-api-access-5gg7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:17.679182 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.679167 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-token" pod "70abeac9-ada5-4b73-8242-5fb3cad8e5a6" (UID: "70abeac9-ada5-4b73-8242-5fb3cad8e5a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:17.774316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.774280 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.774316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.774311 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.774316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.774322 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.774555 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.774330 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.774555 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.774339 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.774555 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.774350 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t25v\" (UniqueName: \"kubernetes.io/projected/409f33be-0503-4f36-ad79-93d1646f51ec-kube-api-access-7t25v\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.774555 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.774359 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.774555 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.774367 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.774555 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.774376 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.774555 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.774385 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.774555 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.774395 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409f33be-0503-4f36-ad79-93d1646f51ec-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.774555 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.774404 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gg7r\" (UniqueName: \"kubernetes.io/projected/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-kube-api-access-5gg7r\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.774555 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.774413 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409f33be-0503-4f36-ad79-93d1646f51ec-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.774555 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.774421 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.774555 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.774429 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.774555 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.774437 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70abeac9-ada5-4b73-8242-5fb3cad8e5a6-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.774555 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.774445 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409f33be-0503-4f36-ad79-93d1646f51ec-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.774555 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.774454 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409f33be-0503-4f36-ad79-93d1646f51ec-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:17.934575 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.934539 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:17.971696 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.971661 5801 generic.go:356] "Generic (PLEG): container finished" podID="70abeac9-ada5-4b73-8242-5fb3cad8e5a6" containerID="8e69e57bf3f8c4647ad767fddd664826b57b23600102c0cd5c9a787f512f920a" exitCode=0 Oct 30 21:50:17.971916 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.971708 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" Oct 30 21:50:17.971916 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.971737 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" event={"ID":"70abeac9-ada5-4b73-8242-5fb3cad8e5a6","Type":"ContainerDied","Data":"8e69e57bf3f8c4647ad767fddd664826b57b23600102c0cd5c9a787f512f920a"} Oct 30 21:50:17.971916 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.971805 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v" event={"ID":"70abeac9-ada5-4b73-8242-5fb3cad8e5a6","Type":"ContainerDied","Data":"c677b155562a5c913f5d0698df7942d976434d7e35a5c446ebe1f308b37d30ee"} Oct 30 21:50:17.971916 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.971827 5801 scope.go:122] "RemoveContainer" containerID="8e69e57bf3f8c4647ad767fddd664826b57b23600102c0cd5c9a787f512f920a" Oct 30 21:50:17.973462 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.973429 5801 generic.go:356] "Generic (PLEG): container finished" podID="a26eafe2-a22a-47a0-ae23-562931b2cead" containerID="84614df4a9236c9a2c63fae4cf0542c00e1e9b6397ceb861b2a3da72f0b70333" exitCode=0 Oct 30 21:50:17.973567 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.973478 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" Oct 30 21:50:17.973567 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.973515 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" event={"ID":"a26eafe2-a22a-47a0-ae23-562931b2cead","Type":"ContainerDied","Data":"84614df4a9236c9a2c63fae4cf0542c00e1e9b6397ceb861b2a3da72f0b70333"} Oct 30 21:50:17.973567 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.973565 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt" event={"ID":"a26eafe2-a22a-47a0-ae23-562931b2cead","Type":"ContainerDied","Data":"a7d0fa460c0c5bdb2e4c5a5895c6e4a535017b1921db4e86e1573f5b25ed2792"} Oct 30 21:50:17.975239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.975213 5801 generic.go:356] "Generic (PLEG): container finished" podID="409f33be-0503-4f36-ad79-93d1646f51ec" containerID="0b29388fe91c2329a24c37d18faade709132c7a3946366de57909f4659b79126" exitCode=0 Oct 30 21:50:17.975362 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.975342 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" Oct 30 21:50:17.975850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.975813 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" event={"ID":"409f33be-0503-4f36-ad79-93d1646f51ec","Type":"ContainerDied","Data":"0b29388fe91c2329a24c37d18faade709132c7a3946366de57909f4659b79126"} Oct 30 21:50:17.975987 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.975863 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt" event={"ID":"409f33be-0503-4f36-ad79-93d1646f51ec","Type":"ContainerDied","Data":"74bce5f3595a76c9f5816df453ebc733526d1764336d4279a442f7dbe64114e4"} Oct 30 21:50:17.990087 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.989545 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz" event={"ID":"2f2c9cba-2588-4e81-9a7c-33fef0ffb58e","Type":"ContainerStarted","Data":"e5f0650bd0a87df55b1b9d31f4fdcb39ba38b15a1d20b9f97f862ed1f81b5e8b"} Oct 30 21:50:17.990087 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.989607 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz" event={"ID":"2f2c9cba-2588-4e81-9a7c-33fef0ffb58e","Type":"ContainerStarted","Data":"87d366d9d3250831bd659147e9f6bf5bff0c4bc028899c5089e24e7d6d907fed"} Oct 30 21:50:17.998189 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.997393 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58c5d82-cm72n" event={"ID":"ce20b749-0d51-4e79-a243-109b758e70f8","Type":"ContainerDied","Data":"aec995878667f36acdcd4041206440f664c1e27d640e26c5ffb5942d2da7ec94"} Oct 30 21:50:17.998189 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.997439 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aec995878667f36acdcd4041206440f664c1e27d640e26c5ffb5942d2da7ec94" Oct 30 21:50:17.998189 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:17.997488 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58c5d82-cm72n" Oct 30 21:50:18.018058 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:18.017926 5801 scope.go:122] "RemoveContainer" containerID="8e69e57bf3f8c4647ad767fddd664826b57b23600102c0cd5c9a787f512f920a" Oct 30 21:50:18.018414 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:50:18.018383 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e69e57bf3f8c4647ad767fddd664826b57b23600102c0cd5c9a787f512f920a\": container with ID starting with 8e69e57bf3f8c4647ad767fddd664826b57b23600102c0cd5c9a787f512f920a not found: ID does not exist" containerID="8e69e57bf3f8c4647ad767fddd664826b57b23600102c0cd5c9a787f512f920a" Oct 30 21:50:18.018546 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:18.018422 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e69e57bf3f8c4647ad767fddd664826b57b23600102c0cd5c9a787f512f920a"} err="failed to get container status \"8e69e57bf3f8c4647ad767fddd664826b57b23600102c0cd5c9a787f512f920a\": rpc error: code = NotFound desc = could not find container \"8e69e57bf3f8c4647ad767fddd664826b57b23600102c0cd5c9a787f512f920a\": container with ID starting with 8e69e57bf3f8c4647ad767fddd664826b57b23600102c0cd5c9a787f512f920a not found: ID does not exist" Oct 30 21:50:18.018546 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:18.018441 5801 scope.go:122] "RemoveContainer" containerID="84614df4a9236c9a2c63fae4cf0542c00e1e9b6397ceb861b2a3da72f0b70333" Oct 30 21:50:18.030927 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:18.030882 5801 scope.go:122] "RemoveContainer" containerID="84614df4a9236c9a2c63fae4cf0542c00e1e9b6397ceb861b2a3da72f0b70333" Oct 30 21:50:18.031605 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:50:18.031445 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84614df4a9236c9a2c63fae4cf0542c00e1e9b6397ceb861b2a3da72f0b70333\": container with ID starting with 84614df4a9236c9a2c63fae4cf0542c00e1e9b6397ceb861b2a3da72f0b70333 not found: ID does not exist" containerID="84614df4a9236c9a2c63fae4cf0542c00e1e9b6397ceb861b2a3da72f0b70333" Oct 30 21:50:18.031605 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:18.031484 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84614df4a9236c9a2c63fae4cf0542c00e1e9b6397ceb861b2a3da72f0b70333"} err="failed to get container status \"84614df4a9236c9a2c63fae4cf0542c00e1e9b6397ceb861b2a3da72f0b70333\": rpc error: code = NotFound desc = could not find container \"84614df4a9236c9a2c63fae4cf0542c00e1e9b6397ceb861b2a3da72f0b70333\": container with ID starting with 84614df4a9236c9a2c63fae4cf0542c00e1e9b6397ceb861b2a3da72f0b70333 not found: ID does not exist" Oct 30 21:50:18.031605 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:18.031509 5801 scope.go:122] "RemoveContainer" containerID="0b29388fe91c2329a24c37d18faade709132c7a3946366de57909f4659b79126" Oct 30 21:50:18.035425 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:18.035377 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz" podStartSLOduration=2.035363271 podStartE2EDuration="2.035363271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:18.018396754 +0000 UTC m=+1286.614213415" watchObservedRunningTime="2026-10-30 21:50:18.035363271 +0000 UTC m=+1286.631179983" Oct 30 21:50:18.036661 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:18.035930 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v"] Oct 30 21:50:18.043707 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:18.043677 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-hzs7v"] Oct 30 21:50:18.046418 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:18.046393 5801 scope.go:122] "RemoveContainer" containerID="0b29388fe91c2329a24c37d18faade709132c7a3946366de57909f4659b79126" Oct 30 21:50:18.046807 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:50:18.046734 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b29388fe91c2329a24c37d18faade709132c7a3946366de57909f4659b79126\": container with ID starting with 0b29388fe91c2329a24c37d18faade709132c7a3946366de57909f4659b79126 not found: ID does not exist" containerID="0b29388fe91c2329a24c37d18faade709132c7a3946366de57909f4659b79126" Oct 30 21:50:18.046869 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:18.046820 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b29388fe91c2329a24c37d18faade709132c7a3946366de57909f4659b79126"} err="failed to get container status \"0b29388fe91c2329a24c37d18faade709132c7a3946366de57909f4659b79126\": rpc error: code = NotFound desc = could not find container \"0b29388fe91c2329a24c37d18faade709132c7a3946366de57909f4659b79126\": container with ID starting with 0b29388fe91c2329a24c37d18faade709132c7a3946366de57909f4659b79126 not found: ID does not exist" Oct 30 21:50:18.063001 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:18.062949 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt"] Oct 30 21:50:18.067020 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:18.066995 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-9gbbt"] Oct 30 21:50:18.080043 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:18.080007 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt"] Oct 30 21:50:18.084176 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:18.084149 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-08aafeac-openshift-default-6cb947dcbf-nkcgt"] Oct 30 21:50:18.684698 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:18.684655 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:19.009411 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:19.009335 5801 generic.go:356] "Generic (PLEG): container finished" podID="2f2c9cba-2588-4e81-9a7c-33fef0ffb58e" containerID="e5f0650bd0a87df55b1b9d31f4fdcb39ba38b15a1d20b9f97f862ed1f81b5e8b" exitCode=0 Oct 30 21:50:19.009551 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:19.009388 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz" event={"ID":"2f2c9cba-2588-4e81-9a7c-33fef0ffb58e","Type":"ContainerDied","Data":"e5f0650bd0a87df55b1b9d31f4fdcb39ba38b15a1d20b9f97f862ed1f81b5e8b"} Oct 30 21:50:19.358370 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:19.358326 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:19.421763 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:19.421724 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt"] Oct 30 21:50:19.505646 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:19.505604 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:19.526371 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:19.526339 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="409f33be-0503-4f36-ad79-93d1646f51ec" path="/var/lib/kubelet/pods/409f33be-0503-4f36-ad79-93d1646f51ec/volumes" Oct 30 21:50:19.527217 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:19.527030 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70abeac9-ada5-4b73-8242-5fb3cad8e5a6" path="/var/lib/kubelet/pods/70abeac9-ada5-4b73-8242-5fb3cad8e5a6/volumes" Oct 30 21:50:19.527578 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:19.527561 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a26eafe2-a22a-47a0-ae23-562931b2cead" path="/var/lib/kubelet/pods/a26eafe2-a22a-47a0-ae23-562931b2cead/volumes" Oct 30 21:50:19.604279 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:19.604242 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:20.177244 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:20.177217 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz" Oct 30 21:50:20.290475 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:20.290389 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:20.310987 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:20.310952 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f2c9cba-2588-4e81-9a7c-33fef0ffb58e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2c9cba-2588-4e81-9a7c-33fef0ffb58e-maas-api-xks-ca\") pod \"2f2c9cba-2588-4e81-9a7c-33fef0ffb58e\" (UID: \"2f2c9cba-2588-4e81-9a7c-33fef0ffb58e\") " Oct 30 21:50:20.311186 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:20.311078 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f2c9cba-2588-4e81-9a7c-33fef0ffb58e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2c9cba-2588-4e81-9a7c-33fef0ffb58e-maas-api-openshift-ca\") pod \"2f2c9cba-2588-4e81-9a7c-33fef0ffb58e\" (UID: \"2f2c9cba-2588-4e81-9a7c-33fef0ffb58e\") " Oct 30 21:50:20.311265 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:20.311213 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f2c9cba-2588-4e81-9a7c-33fef0ffb58e-kube-api-access-kdx5p\" (UniqueName: \"kubernetes.io/projected/2f2c9cba-2588-4e81-9a7c-33fef0ffb58e-kube-api-access-kdx5p\") pod \"2f2c9cba-2588-4e81-9a7c-33fef0ffb58e\" (UID: \"2f2c9cba-2588-4e81-9a7c-33fef0ffb58e\") " Oct 30 21:50:20.311494 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:20.311463 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f2c9cba-2588-4e81-9a7c-33fef0ffb58e-maas-api-openshift-ca" pod "2f2c9cba-2588-4e81-9a7c-33fef0ffb58e" (UID: "2f2c9cba-2588-4e81-9a7c-33fef0ffb58e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:20.311673 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:20.311658 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2c9cba-2588-4e81-9a7c-33fef0ffb58e-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:20.313210 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:20.313182 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f2c9cba-2588-4e81-9a7c-33fef0ffb58e-maas-api-xks-ca" pod "2f2c9cba-2588-4e81-9a7c-33fef0ffb58e" (UID: "2f2c9cba-2588-4e81-9a7c-33fef0ffb58e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:20.313318 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:20.313247 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f2c9cba-2588-4e81-9a7c-33fef0ffb58e-kube-api-access-kdx5p" pod "2f2c9cba-2588-4e81-9a7c-33fef0ffb58e" (UID: "2f2c9cba-2588-4e81-9a7c-33fef0ffb58e"). InnerVolumeSpecName "kube-api-access-kdx5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:20.415084 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:20.415029 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2c9cba-2588-4e81-9a7c-33fef0ffb58e-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:20.415084 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:20.415068 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdx5p\" (UniqueName: \"kubernetes.io/projected/2f2c9cba-2588-4e81-9a7c-33fef0ffb58e-kube-api-access-kdx5p\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:20.970647 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:20.970608 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:21.044029 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:21.043985 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz" Oct 30 21:50:21.044310 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:21.043985 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz" event={"ID":"2f2c9cba-2588-4e81-9a7c-33fef0ffb58e","Type":"ContainerDied","Data":"87d366d9d3250831bd659147e9f6bf5bff0c4bc028899c5089e24e7d6d907fed"} Oct 30 21:50:21.044310 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:21.044122 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87d366d9d3250831bd659147e9f6bf5bff0c4bc028899c5089e24e7d6d907fed" Oct 30 21:50:21.550682 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:21.550645 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:21.839562 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:21.839520 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:21.876682 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:21.876642 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-950784-7rs77"] Oct 30 21:50:21.880623 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:21.880596 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-950784-7rs77"] Oct 30 21:50:21.939120 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:21.939079 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:22.051026 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:22.050985 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:22.581004 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:22.580958 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:23.240396 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:23.240357 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:23.526929 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:23.526848 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def78ea4-463d-4580-9a63-c1f9be5507f9" path="/var/lib/kubelet/pods/def78ea4-463d-4580-9a63-c1f9be5507f9/volumes" Oct 30 21:50:23.979519 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:23.979486 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9"] Oct 30 21:50:23.981442 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:23.981413 5801 cpu_manager.go:414] "containerMap: removing container" podUID="409f33be-0503-4f36-ad79-93d1646f51ec" containerName="istio-proxy" Oct 30 21:50:23.981627 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:23.981611 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="409f33be-0503-4f36-ad79-93d1646f51ec" containerName="istio-proxy" Oct 30 21:50:23.981775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:23.981746 5801 cpu_manager.go:414] "containerMap: removing container" podUID="a26eafe2-a22a-47a0-ae23-562931b2cead" containerName="istio-proxy" Oct 30 21:50:23.981861 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:23.981850 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a26eafe2-a22a-47a0-ae23-562931b2cead" containerName="istio-proxy" Oct 30 21:50:23.981940 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:23.981932 5801 cpu_manager.go:414] "containerMap: removing container" podUID="70abeac9-ada5-4b73-8242-5fb3cad8e5a6" containerName="istio-proxy" Oct 30 21:50:23.982002 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:23.981993 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70abeac9-ada5-4b73-8242-5fb3cad8e5a6" containerName="istio-proxy" Oct 30 21:50:23.982096 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:23.982086 5801 cpu_manager.go:414] "containerMap: removing container" podUID="ce20b749-0d51-4e79-a243-109b758e70f8" containerName="revoke-keys" Oct 30 21:50:23.982188 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:23.982177 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce20b749-0d51-4e79-a243-109b758e70f8" containerName="revoke-keys" Oct 30 21:50:23.982262 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:23.982253 5801 cpu_manager.go:414] "containerMap: removing container" podUID="2f2c9cba-2588-4e81-9a7c-33fef0ffb58e" containerName="revoke-keys" Oct 30 21:50:23.982328 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:23.982318 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2c9cba-2588-4e81-9a7c-33fef0ffb58e" containerName="revoke-keys" Oct 30 21:50:23.982575 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:23.982559 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="409f33be-0503-4f36-ad79-93d1646f51ec" containerName="istio-proxy" Oct 30 21:50:23.982700 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:23.982689 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="a26eafe2-a22a-47a0-ae23-562931b2cead" containerName="istio-proxy" Oct 30 21:50:23.982811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:23.982800 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="70abeac9-ada5-4b73-8242-5fb3cad8e5a6" containerName="istio-proxy" Oct 30 21:50:23.982910 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:23.982900 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce20b749-0d51-4e79-a243-109b758e70f8" containerName="revoke-keys" Oct 30 21:50:23.982991 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:23.982982 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f2c9cba-2588-4e81-9a7c-33fef0ffb58e" containerName="revoke-keys" Oct 30 21:50:23.988359 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:23.988337 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9"] Oct 30 21:50:23.988447 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:23.988428 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9" Oct 30 21:50:23.992138 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:23.992106 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:50:23.993991 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:23.993860 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:50:23.993991 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:23.993940 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:50:23.993991 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:23.993958 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:50:24.095332 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:24.095295 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3658bb0a-6bcf-47a4-a1f9-fff73c104c17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9\" (UID: \"3658bb0a-6bcf-47a4-a1f9-fff73c104c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9" Oct 30 21:50:24.095513 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:24.095339 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtb8f\" (UniqueName: \"kubernetes.io/projected/3658bb0a-6bcf-47a4-a1f9-fff73c104c17-kube-api-access-dtb8f\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9\" (UID: \"3658bb0a-6bcf-47a4-a1f9-fff73c104c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9" Oct 30 21:50:24.095621 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:24.095594 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3658bb0a-6bcf-47a4-a1f9-fff73c104c17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9\" (UID: \"3658bb0a-6bcf-47a4-a1f9-fff73c104c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9" Oct 30 21:50:24.130210 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:24.130173 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:24.199776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:24.197072 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3658bb0a-6bcf-47a4-a1f9-fff73c104c17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9\" (UID: \"3658bb0a-6bcf-47a4-a1f9-fff73c104c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9" Oct 30 21:50:24.199776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:24.197126 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtb8f\" (UniqueName: \"kubernetes.io/projected/3658bb0a-6bcf-47a4-a1f9-fff73c104c17-kube-api-access-dtb8f\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9\" (UID: \"3658bb0a-6bcf-47a4-a1f9-fff73c104c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9" Oct 30 21:50:24.199776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:24.197305 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3658bb0a-6bcf-47a4-a1f9-fff73c104c17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9\" (UID: \"3658bb0a-6bcf-47a4-a1f9-fff73c104c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9" Oct 30 21:50:24.212309 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:24.212275 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3658bb0a-6bcf-47a4-a1f9-fff73c104c17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9\" (UID: \"3658bb0a-6bcf-47a4-a1f9-fff73c104c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9" Oct 30 21:50:24.213251 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:24.213224 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3658bb0a-6bcf-47a4-a1f9-fff73c104c17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9\" (UID: \"3658bb0a-6bcf-47a4-a1f9-fff73c104c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9" Oct 30 21:50:24.224569 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:24.224539 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtb8f\" (UniqueName: \"kubernetes.io/projected/3658bb0a-6bcf-47a4-a1f9-fff73c104c17-kube-api-access-dtb8f\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9\" (UID: \"3658bb0a-6bcf-47a4-a1f9-fff73c104c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9" Oct 30 21:50:24.307315 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:24.307290 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9" Oct 30 21:50:24.422775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:24.422614 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt" podUID="f1db92e5-5f7e-42ee-b389-ac3690448d17" containerName="payload-processing" containerID="cri-o://93cd41f9752099fe0030a4393836e638a20b73e7bd170b5f88bcb7af4f200ba6" gracePeriod=25 Oct 30 21:50:24.641399 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:24.641359 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9"] Oct 30 21:50:24.721171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:24.721148 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt" Oct 30 21:50:24.805698 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:24.805667 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1db92e5-5f7e-42ee-b389-ac3690448d17-kube-api-access-95cj2\" (UniqueName: \"kubernetes.io/projected/f1db92e5-5f7e-42ee-b389-ac3690448d17-kube-api-access-95cj2\") pod \"f1db92e5-5f7e-42ee-b389-ac3690448d17\" (UID: \"f1db92e5-5f7e-42ee-b389-ac3690448d17\") " Oct 30 21:50:24.805889 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:24.805718 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1db92e5-5f7e-42ee-b389-ac3690448d17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1db92e5-5f7e-42ee-b389-ac3690448d17-plugins-config-volume\") pod \"f1db92e5-5f7e-42ee-b389-ac3690448d17\" (UID: \"f1db92e5-5f7e-42ee-b389-ac3690448d17\") " Oct 30 21:50:24.806226 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:24.806200 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1db92e5-5f7e-42ee-b389-ac3690448d17-plugins-config-volume" pod "f1db92e5-5f7e-42ee-b389-ac3690448d17" (UID: "f1db92e5-5f7e-42ee-b389-ac3690448d17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:24.806589 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:24.806565 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1db92e5-5f7e-42ee-b389-ac3690448d17-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:24.887266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:24.887186 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1db92e5-5f7e-42ee-b389-ac3690448d17-kube-api-access-95cj2" pod "f1db92e5-5f7e-42ee-b389-ac3690448d17" (UID: "f1db92e5-5f7e-42ee-b389-ac3690448d17"). InnerVolumeSpecName "kube-api-access-95cj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:24.889359 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:24.889336 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl"] Oct 30 21:50:24.889536 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:24.889516 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" podUID="c1ee1395-263a-4afa-8f55-244a8edcd3a8" containerName="istio-proxy" containerID="cri-o://c136f0caf0208ce9494c36ce7c024a1280db31ec7ff6130258c7bde4b55e827a" gracePeriod=30 Oct 30 21:50:24.908789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:24.908747 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95cj2\" (UniqueName: \"kubernetes.io/projected/f1db92e5-5f7e-42ee-b389-ac3690448d17-kube-api-access-95cj2\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:25.096802 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:25.096748 5801 generic.go:356] "Generic (PLEG): container finished" podID="f1db92e5-5f7e-42ee-b389-ac3690448d17" containerID="93cd41f9752099fe0030a4393836e638a20b73e7bd170b5f88bcb7af4f200ba6" exitCode=0 Oct 30 21:50:25.097301 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:25.096851 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt" Oct 30 21:50:25.097301 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:25.096921 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt" event={"ID":"f1db92e5-5f7e-42ee-b389-ac3690448d17","Type":"ContainerDied","Data":"93cd41f9752099fe0030a4393836e638a20b73e7bd170b5f88bcb7af4f200ba6"} Oct 30 21:50:25.097301 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:25.096963 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt" event={"ID":"f1db92e5-5f7e-42ee-b389-ac3690448d17","Type":"ContainerDied","Data":"0d0e622ee5fab1b9b8120d136135acfc7a6a3317cc5cb0e002ad4c204787f4c2"} Oct 30 21:50:25.097301 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:25.096987 5801 scope.go:122] "RemoveContainer" containerID="93cd41f9752099fe0030a4393836e638a20b73e7bd170b5f88bcb7af4f200ba6" Oct 30 21:50:25.099314 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:25.099286 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9" event={"ID":"3658bb0a-6bcf-47a4-a1f9-fff73c104c17","Type":"ContainerStarted","Data":"faaa1e75787b360f0c15c5880c38b9906a67fa44de248ef29b00b89707fd8d50"} Oct 30 21:50:25.099438 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:25.099331 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9" event={"ID":"3658bb0a-6bcf-47a4-a1f9-fff73c104c17","Type":"ContainerStarted","Data":"f8d0231946d54608595a78a3123b172c2b6e407858a316d7c3691d3a87903720"} Oct 30 21:50:25.108261 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:25.108241 5801 scope.go:122] "RemoveContainer" containerID="93cd41f9752099fe0030a4393836e638a20b73e7bd170b5f88bcb7af4f200ba6" Oct 30 21:50:25.108521 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:50:25.108498 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93cd41f9752099fe0030a4393836e638a20b73e7bd170b5f88bcb7af4f200ba6\": container with ID starting with 93cd41f9752099fe0030a4393836e638a20b73e7bd170b5f88bcb7af4f200ba6 not found: ID does not exist" containerID="93cd41f9752099fe0030a4393836e638a20b73e7bd170b5f88bcb7af4f200ba6" Oct 30 21:50:25.108564 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:25.108530 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93cd41f9752099fe0030a4393836e638a20b73e7bd170b5f88bcb7af4f200ba6"} err="failed to get container status \"93cd41f9752099fe0030a4393836e638a20b73e7bd170b5f88bcb7af4f200ba6\": rpc error: code = NotFound desc = could not find container \"93cd41f9752099fe0030a4393836e638a20b73e7bd170b5f88bcb7af4f200ba6\": container with ID starting with 93cd41f9752099fe0030a4393836e638a20b73e7bd170b5f88bcb7af4f200ba6 not found: ID does not exist" Oct 30 21:50:25.124931 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:25.124888 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9" podStartSLOduration=2.124876048 podStartE2EDuration="2.124876048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:25.119217946 +0000 UTC m=+1293.715034606" watchObservedRunningTime="2026-10-30 21:50:25.124876048 +0000 UTC m=+1293.720692707" Oct 30 21:50:25.138641 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:25.138569 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt"] Oct 30 21:50:25.146786 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:25.146736 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1f725d-64b84964f5-np9zt"] Oct 30 21:50:25.249582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:25.249530 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:25.526017 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:25.525937 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1db92e5-5f7e-42ee-b389-ac3690448d17" path="/var/lib/kubelet/pods/f1db92e5-5f7e-42ee-b389-ac3690448d17/volumes" Oct 30 21:50:25.937916 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:25.937881 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:26.109429 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:26.109396 5801 generic.go:356] "Generic (PLEG): container finished" podID="3658bb0a-6bcf-47a4-a1f9-fff73c104c17" containerID="faaa1e75787b360f0c15c5880c38b9906a67fa44de248ef29b00b89707fd8d50" exitCode=0 Oct 30 21:50:26.109900 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:26.109472 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9" event={"ID":"3658bb0a-6bcf-47a4-a1f9-fff73c104c17","Type":"ContainerDied","Data":"faaa1e75787b360f0c15c5880c38b9906a67fa44de248ef29b00b89707fd8d50"} Oct 30 21:50:26.490352 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:26.490002 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j"] Oct 30 21:50:26.504251 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:26.504216 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl"] Oct 30 21:50:26.587941 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:26.587690 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:26.688430 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:26.688385 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:26.788245 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:26.788135 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:26.852533 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:26.852489 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp"] Oct 30 21:50:26.888779 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:26.881440 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1f725d-prphp"] Oct 30 21:50:26.894774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:26.891294 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:26.999457 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:26.999420 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:27.385537 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:27.385464 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:27.395740 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:27.395720 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9" Oct 30 21:50:27.447958 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:27.447923 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3658bb0a-6bcf-47a4-a1f9-fff73c104c17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3658bb0a-6bcf-47a4-a1f9-fff73c104c17-maas-api-xks-ca\") pod \"3658bb0a-6bcf-47a4-a1f9-fff73c104c17\" (UID: \"3658bb0a-6bcf-47a4-a1f9-fff73c104c17\") " Oct 30 21:50:27.448181 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:27.448156 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3658bb0a-6bcf-47a4-a1f9-fff73c104c17-kube-api-access-dtb8f\" (UniqueName: \"kubernetes.io/projected/3658bb0a-6bcf-47a4-a1f9-fff73c104c17-kube-api-access-dtb8f\") pod \"3658bb0a-6bcf-47a4-a1f9-fff73c104c17\" (UID: \"3658bb0a-6bcf-47a4-a1f9-fff73c104c17\") " Oct 30 21:50:27.448311 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:27.448294 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3658bb0a-6bcf-47a4-a1f9-fff73c104c17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3658bb0a-6bcf-47a4-a1f9-fff73c104c17-maas-api-openshift-ca\") pod \"3658bb0a-6bcf-47a4-a1f9-fff73c104c17\" (UID: \"3658bb0a-6bcf-47a4-a1f9-fff73c104c17\") " Oct 30 21:50:27.448697 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:27.448663 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3658bb0a-6bcf-47a4-a1f9-fff73c104c17-maas-api-openshift-ca" pod "3658bb0a-6bcf-47a4-a1f9-fff73c104c17" (UID: "3658bb0a-6bcf-47a4-a1f9-fff73c104c17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:27.448876 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:27.448852 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3658bb0a-6bcf-47a4-a1f9-fff73c104c17-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:27.450171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:27.450146 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3658bb0a-6bcf-47a4-a1f9-fff73c104c17-maas-api-xks-ca" pod "3658bb0a-6bcf-47a4-a1f9-fff73c104c17" (UID: "3658bb0a-6bcf-47a4-a1f9-fff73c104c17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:27.450835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:27.450812 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3658bb0a-6bcf-47a4-a1f9-fff73c104c17-kube-api-access-dtb8f" pod "3658bb0a-6bcf-47a4-a1f9-fff73c104c17" (UID: "3658bb0a-6bcf-47a4-a1f9-fff73c104c17"). InnerVolumeSpecName "kube-api-access-dtb8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:27.530397 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:27.530360 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55166a7a-d549-401f-bc28-3f90b8774a59" path="/var/lib/kubelet/pods/55166a7a-d549-401f-bc28-3f90b8774a59/volumes" Oct 30 21:50:27.550396 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:27.550372 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3658bb0a-6bcf-47a4-a1f9-fff73c104c17-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:27.550396 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:27.550397 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtb8f\" (UniqueName: \"kubernetes.io/projected/3658bb0a-6bcf-47a4-a1f9-fff73c104c17-kube-api-access-dtb8f\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:27.790104 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:27.790019 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:28.142922 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:28.142888 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9" Oct 30 21:50:28.142922 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:28.142920 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9" event={"ID":"3658bb0a-6bcf-47a4-a1f9-fff73c104c17","Type":"ContainerDied","Data":"f8d0231946d54608595a78a3123b172c2b6e407858a316d7c3691d3a87903720"} Oct 30 21:50:28.143165 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:28.142972 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8d0231946d54608595a78a3123b172c2b6e407858a316d7c3691d3a87903720" Oct 30 21:50:28.447513 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:28.447412 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:29.115178 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:29.115152 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:29.641410 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:29.641364 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:29.748761 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:29.748711 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:29.998441 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:29.998407 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:30.126467 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.126434 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5"] Oct 30 21:50:30.126704 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.126674 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" podUID="f9975b4c-127e-4130-ae64-b8785b0acc7f" containerName="istio-proxy" containerID="cri-o://6413883aa622617e7e9324d67928ea69d7313b63d4be87ca5628bf65627aabc8" gracePeriod=30 Oct 30 21:50:30.165894 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.165787 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:30.200468 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.200432 5801 generic.go:356] "Generic (PLEG): container finished" podID="c1ee1395-263a-4afa-8f55-244a8edcd3a8" containerID="c136f0caf0208ce9494c36ce7c024a1280db31ec7ff6130258c7bde4b55e827a" exitCode=0 Oct 30 21:50:30.200603 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.200501 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" event={"ID":"c1ee1395-263a-4afa-8f55-244a8edcd3a8","Type":"ContainerDied","Data":"c136f0caf0208ce9494c36ce7c024a1280db31ec7ff6130258c7bde4b55e827a"} Oct 30 21:50:30.200603 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.200556 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" event={"ID":"c1ee1395-263a-4afa-8f55-244a8edcd3a8","Type":"ContainerDied","Data":"aed51a9989fb7a02f03c8e420c8485cf2328ee4d8013d560b4a4edd358b13fd8"} Oct 30 21:50:30.200603 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.200575 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aed51a9989fb7a02f03c8e420c8485cf2328ee4d8013d560b4a4edd358b13fd8" Oct 30 21:50:30.279830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.276311 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:50:30.362456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.342240 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-workload-socket\") pod \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " Oct 30 21:50:30.362456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.342346 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-data\") pod \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " Oct 30 21:50:30.362456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.342404 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-workload-certs\") pod \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " Oct 30 21:50:30.362456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.342485 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-credential-socket\") pod \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " Oct 30 21:50:30.362456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.342530 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-envoy\") pod \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " Oct 30 21:50:30.362456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.342621 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ee1395-263a-4afa-8f55-244a8edcd3a8-kube-api-access-dw775\" (UniqueName: \"kubernetes.io/projected/c1ee1395-263a-4afa-8f55-244a8edcd3a8-kube-api-access-dw775\") pod \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " Oct 30 21:50:30.362456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.342727 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-podinfo\") pod \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " Oct 30 21:50:30.362456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.342812 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-token\" (UniqueName: \"kubernetes.io/projected/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-token\") pod \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " Oct 30 21:50:30.362456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.342874 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istiod-ca-cert\") pod \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\" (UID: \"c1ee1395-263a-4afa-8f55-244a8edcd3a8\") " Oct 30 21:50:30.362456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.354985 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-workload-certs" pod "c1ee1395-263a-4afa-8f55-244a8edcd3a8" (UID: "c1ee1395-263a-4afa-8f55-244a8edcd3a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:30.362456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.355249 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-workload-socket" pod "c1ee1395-263a-4afa-8f55-244a8edcd3a8" (UID: "c1ee1395-263a-4afa-8f55-244a8edcd3a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:30.362456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.355425 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-data" pod "c1ee1395-263a-4afa-8f55-244a8edcd3a8" (UID: "c1ee1395-263a-4afa-8f55-244a8edcd3a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:30.362456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.362319 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ee1395-263a-4afa-8f55-244a8edcd3a8-kube-api-access-dw775" pod "c1ee1395-263a-4afa-8f55-244a8edcd3a8" (UID: "c1ee1395-263a-4afa-8f55-244a8edcd3a8"). InnerVolumeSpecName "kube-api-access-dw775". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:30.364128 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.363471 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-credential-socket" pod "c1ee1395-263a-4afa-8f55-244a8edcd3a8" (UID: "c1ee1395-263a-4afa-8f55-244a8edcd3a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:30.364717 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.364609 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istiod-ca-cert" pod "c1ee1395-263a-4afa-8f55-244a8edcd3a8" (UID: "c1ee1395-263a-4afa-8f55-244a8edcd3a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:30.364717 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.364687 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-envoy" pod "c1ee1395-263a-4afa-8f55-244a8edcd3a8" (UID: "c1ee1395-263a-4afa-8f55-244a8edcd3a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:30.367634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.365208 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:30.367634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.365229 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:30.367634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.365244 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:30.367634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.365259 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:30.367634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.365273 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:30.367634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.365287 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw775\" (UniqueName: \"kubernetes.io/projected/c1ee1395-263a-4afa-8f55-244a8edcd3a8-kube-api-access-dw775\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:30.367634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.365301 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:30.404811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.387769 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-podinfo" pod "c1ee1395-263a-4afa-8f55-244a8edcd3a8" (UID: "c1ee1395-263a-4afa-8f55-244a8edcd3a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:30.404811 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:50:30.387955 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:50:30.404811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.390475 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-token" pod "c1ee1395-263a-4afa-8f55-244a8edcd3a8" (UID: "c1ee1395-263a-4afa-8f55-244a8edcd3a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:30.467370 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.467294 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:30.467370 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.467323 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ee1395-263a-4afa-8f55-244a8edcd3a8-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:30.580610 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.580576 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm"] Oct 30 21:50:30.581231 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.581215 5801 cpu_manager.go:414] "containerMap: removing container" podUID="f1db92e5-5f7e-42ee-b389-ac3690448d17" containerName="payload-processing" Oct 30 21:50:30.581304 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.581233 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1db92e5-5f7e-42ee-b389-ac3690448d17" containerName="payload-processing" Oct 30 21:50:30.581304 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.581254 5801 cpu_manager.go:414] "containerMap: removing container" podUID="3658bb0a-6bcf-47a4-a1f9-fff73c104c17" containerName="revoke-keys" Oct 30 21:50:30.581304 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.581259 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3658bb0a-6bcf-47a4-a1f9-fff73c104c17" containerName="revoke-keys" Oct 30 21:50:30.581304 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.581272 5801 cpu_manager.go:414] "containerMap: removing container" podUID="c1ee1395-263a-4afa-8f55-244a8edcd3a8" containerName="istio-proxy" Oct 30 21:50:30.581304 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.581278 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1ee1395-263a-4afa-8f55-244a8edcd3a8" containerName="istio-proxy" Oct 30 21:50:30.581530 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.581383 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1db92e5-5f7e-42ee-b389-ac3690448d17" containerName="payload-processing" Oct 30 21:50:30.581530 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.581392 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="3658bb0a-6bcf-47a4-a1f9-fff73c104c17" containerName="revoke-keys" Oct 30 21:50:30.581530 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.581401 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1ee1395-263a-4afa-8f55-244a8edcd3a8" containerName="istio-proxy" Oct 30 21:50:30.586064 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.586041 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm" Oct 30 21:50:30.590901 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.590860 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:50:30.591045 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.590974 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:50:30.597195 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.597173 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm"] Oct 30 21:50:30.668324 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.668286 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:50:30.668324 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.668286 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:50:30.670049 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.670026 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d28203f-5410-4c0e-b475-a0a4ab6afe74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm\" (UID: \"7d28203f-5410-4c0e-b475-a0a4ab6afe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm" Oct 30 21:50:30.670137 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.670096 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvgx2\" (UniqueName: \"kubernetes.io/projected/7d28203f-5410-4c0e-b475-a0a4ab6afe74-kube-api-access-qvgx2\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm\" (UID: \"7d28203f-5410-4c0e-b475-a0a4ab6afe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm" Oct 30 21:50:30.670199 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.670179 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d28203f-5410-4c0e-b475-a0a4ab6afe74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm\" (UID: \"7d28203f-5410-4c0e-b475-a0a4ab6afe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm" Oct 30 21:50:30.771305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.771218 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvgx2\" (UniqueName: \"kubernetes.io/projected/7d28203f-5410-4c0e-b475-a0a4ab6afe74-kube-api-access-qvgx2\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm\" (UID: \"7d28203f-5410-4c0e-b475-a0a4ab6afe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm" Oct 30 21:50:30.771305 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.771264 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d28203f-5410-4c0e-b475-a0a4ab6afe74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm\" (UID: \"7d28203f-5410-4c0e-b475-a0a4ab6afe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm" Oct 30 21:50:30.771633 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.771609 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d28203f-5410-4c0e-b475-a0a4ab6afe74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm\" (UID: \"7d28203f-5410-4c0e-b475-a0a4ab6afe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm" Oct 30 21:50:30.772321 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.772298 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d28203f-5410-4c0e-b475-a0a4ab6afe74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm\" (UID: \"7d28203f-5410-4c0e-b475-a0a4ab6afe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm" Oct 30 21:50:30.773742 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.773723 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d28203f-5410-4c0e-b475-a0a4ab6afe74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm\" (UID: \"7d28203f-5410-4c0e-b475-a0a4ab6afe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm" Oct 30 21:50:30.783465 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.783439 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvgx2\" (UniqueName: \"kubernetes.io/projected/7d28203f-5410-4c0e-b475-a0a4ab6afe74-kube-api-access-qvgx2\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm\" (UID: \"7d28203f-5410-4c0e-b475-a0a4ab6afe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm" Oct 30 21:50:30.905313 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.905277 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm" Oct 30 21:50:30.993179 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:30.993085 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:31.195480 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:31.195451 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm"] Oct 30 21:50:31.196887 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:50:31.196864 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d28203f_5410_4c0e_b475_a0a4ab6afe74.slice/crio-1653f25a2a143dd29b77936cc6765f3f4165044d3663a57d08a8199b9636ec10 WatchSource:0}: Error finding container 1653f25a2a143dd29b77936cc6765f3f4165044d3663a57d08a8199b9636ec10: Status 404 returned error can't find the container with id 1653f25a2a143dd29b77936cc6765f3f4165044d3663a57d08a8199b9636ec10 Oct 30 21:50:31.210386 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:31.210357 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm" event={"ID":"7d28203f-5410-4c0e-b475-a0a4ab6afe74","Type":"ContainerStarted","Data":"1653f25a2a143dd29b77936cc6765f3f4165044d3663a57d08a8199b9636ec10"} Oct 30 21:50:31.210572 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:31.210555 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl" Oct 30 21:50:31.263806 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:31.263775 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl"] Oct 30 21:50:31.270548 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:31.270512 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-s92bl"] Oct 30 21:50:31.490537 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:31.490450 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j" podUID="d37caf73-c0d3-4433-8877-ee88415dd6c6" containerName="payload-processing" containerID="cri-o://66618bf5f4000e93e6c657e7a3ffe6b8308f2a29aa99e02647ea636e195d07db" gracePeriod=25 Oct 30 21:50:31.504573 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:31.504536 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl" podUID="8be262d1-439b-4a2e-8658-e3ae067edc61" containerName="payload-pre-processing" containerID="cri-o://f35b82c230a040af87fdcc775e5677097ef8e85f67f31d908db994890601a9e1" gracePeriod=25 Oct 30 21:50:31.528857 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:31.528821 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1ee1395-263a-4afa-8f55-244a8edcd3a8" path="/var/lib/kubelet/pods/c1ee1395-263a-4afa-8f55-244a8edcd3a8/volumes" Oct 30 21:50:31.686227 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:31.685415 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:31.783794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:31.779727 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl" Oct 30 21:50:31.872492 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:31.872461 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j" Oct 30 21:50:31.918647 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:31.918611 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be262d1-439b-4a2e-8658-e3ae067edc61-kube-api-access-4zdkb\" (UniqueName: \"kubernetes.io/projected/8be262d1-439b-4a2e-8658-e3ae067edc61-kube-api-access-4zdkb\") pod \"8be262d1-439b-4a2e-8658-e3ae067edc61\" (UID: \"8be262d1-439b-4a2e-8658-e3ae067edc61\") " Oct 30 21:50:31.918882 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:31.918692 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8be262d1-439b-4a2e-8658-e3ae067edc61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be262d1-439b-4a2e-8658-e3ae067edc61-plugins-config-volume\") pod \"8be262d1-439b-4a2e-8658-e3ae067edc61\" (UID: \"8be262d1-439b-4a2e-8658-e3ae067edc61\") " Oct 30 21:50:31.919160 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:31.919128 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be262d1-439b-4a2e-8658-e3ae067edc61-plugins-config-volume" pod "8be262d1-439b-4a2e-8658-e3ae067edc61" (UID: "8be262d1-439b-4a2e-8658-e3ae067edc61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:31.919389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:31.919356 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be262d1-439b-4a2e-8658-e3ae067edc61-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:31.920932 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:31.920903 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be262d1-439b-4a2e-8658-e3ae067edc61-kube-api-access-4zdkb" pod "8be262d1-439b-4a2e-8658-e3ae067edc61" (UID: "8be262d1-439b-4a2e-8658-e3ae067edc61"). InnerVolumeSpecName "kube-api-access-4zdkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:32.020739 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.020643 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d37caf73-c0d3-4433-8877-ee88415dd6c6-kube-api-access-srnqd\" (UniqueName: \"kubernetes.io/projected/d37caf73-c0d3-4433-8877-ee88415dd6c6-kube-api-access-srnqd\") pod \"d37caf73-c0d3-4433-8877-ee88415dd6c6\" (UID: \"d37caf73-c0d3-4433-8877-ee88415dd6c6\") " Oct 30 21:50:32.020946 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.020813 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d37caf73-c0d3-4433-8877-ee88415dd6c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37caf73-c0d3-4433-8877-ee88415dd6c6-plugins-config-volume\") pod \"d37caf73-c0d3-4433-8877-ee88415dd6c6\" (UID: \"d37caf73-c0d3-4433-8877-ee88415dd6c6\") " Oct 30 21:50:32.021266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.021237 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d37caf73-c0d3-4433-8877-ee88415dd6c6-plugins-config-volume" pod "d37caf73-c0d3-4433-8877-ee88415dd6c6" (UID: "d37caf73-c0d3-4433-8877-ee88415dd6c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:32.021537 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.021511 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zdkb\" (UniqueName: \"kubernetes.io/projected/8be262d1-439b-4a2e-8658-e3ae067edc61-kube-api-access-4zdkb\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:32.021537 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.021539 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37caf73-c0d3-4433-8877-ee88415dd6c6-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:32.023135 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.023109 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37caf73-c0d3-4433-8877-ee88415dd6c6-kube-api-access-srnqd" pod "d37caf73-c0d3-4433-8877-ee88415dd6c6" (UID: "d37caf73-c0d3-4433-8877-ee88415dd6c6"). InnerVolumeSpecName "kube-api-access-srnqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:32.122492 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.122456 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srnqd\" (UniqueName: \"kubernetes.io/projected/d37caf73-c0d3-4433-8877-ee88415dd6c6-kube-api-access-srnqd\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:32.222947 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.222914 5801 generic.go:356] "Generic (PLEG): container finished" podID="7d28203f-5410-4c0e-b475-a0a4ab6afe74" containerID="2a349429c74c8f5062646a6a61c97a059f6a34358566863653562fb916797dfc" exitCode=0 Oct 30 21:50:32.223112 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.223011 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm" event={"ID":"7d28203f-5410-4c0e-b475-a0a4ab6afe74","Type":"ContainerDied","Data":"2a349429c74c8f5062646a6a61c97a059f6a34358566863653562fb916797dfc"} Oct 30 21:50:32.224545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.224522 5801 generic.go:356] "Generic (PLEG): container finished" podID="8be262d1-439b-4a2e-8658-e3ae067edc61" containerID="f35b82c230a040af87fdcc775e5677097ef8e85f67f31d908db994890601a9e1" exitCode=0 Oct 30 21:50:32.224666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.224555 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl" event={"ID":"8be262d1-439b-4a2e-8658-e3ae067edc61","Type":"ContainerDied","Data":"f35b82c230a040af87fdcc775e5677097ef8e85f67f31d908db994890601a9e1"} Oct 30 21:50:32.224666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.224578 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl" Oct 30 21:50:32.224666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.224608 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl" event={"ID":"8be262d1-439b-4a2e-8658-e3ae067edc61","Type":"ContainerDied","Data":"54876a52528427e07524c38164deb899c7898f66b1165c5da46ac9a1050659db"} Oct 30 21:50:32.224666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.224640 5801 scope.go:122] "RemoveContainer" containerID="f35b82c230a040af87fdcc775e5677097ef8e85f67f31d908db994890601a9e1" Oct 30 21:50:32.226218 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.226189 5801 generic.go:356] "Generic (PLEG): container finished" podID="d37caf73-c0d3-4433-8877-ee88415dd6c6" containerID="66618bf5f4000e93e6c657e7a3ffe6b8308f2a29aa99e02647ea636e195d07db" exitCode=0 Oct 30 21:50:32.226330 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.226222 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j" event={"ID":"d37caf73-c0d3-4433-8877-ee88415dd6c6","Type":"ContainerDied","Data":"66618bf5f4000e93e6c657e7a3ffe6b8308f2a29aa99e02647ea636e195d07db"} Oct 30 21:50:32.226330 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.226236 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j" Oct 30 21:50:32.226330 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.226267 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j" event={"ID":"d37caf73-c0d3-4433-8877-ee88415dd6c6","Type":"ContainerDied","Data":"a2b98641a775fa5ffa0006c776cc042ee1874ab05e8cb478b889982ee480962f"} Oct 30 21:50:32.249209 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.244907 5801 scope.go:122] "RemoveContainer" containerID="f35b82c230a040af87fdcc775e5677097ef8e85f67f31d908db994890601a9e1" Oct 30 21:50:32.249209 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:50:32.245225 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f35b82c230a040af87fdcc775e5677097ef8e85f67f31d908db994890601a9e1\": container with ID starting with f35b82c230a040af87fdcc775e5677097ef8e85f67f31d908db994890601a9e1 not found: ID does not exist" containerID="f35b82c230a040af87fdcc775e5677097ef8e85f67f31d908db994890601a9e1" Oct 30 21:50:32.249209 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.245256 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f35b82c230a040af87fdcc775e5677097ef8e85f67f31d908db994890601a9e1"} err="failed to get container status \"f35b82c230a040af87fdcc775e5677097ef8e85f67f31d908db994890601a9e1\": rpc error: code = NotFound desc = could not find container \"f35b82c230a040af87fdcc775e5677097ef8e85f67f31d908db994890601a9e1\": container with ID starting with f35b82c230a040af87fdcc775e5677097ef8e85f67f31d908db994890601a9e1 not found: ID does not exist" Oct 30 21:50:32.249209 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.245276 5801 scope.go:122] "RemoveContainer" containerID="66618bf5f4000e93e6c657e7a3ffe6b8308f2a29aa99e02647ea636e195d07db" Oct 30 21:50:32.261088 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.261053 5801 scope.go:122] "RemoveContainer" containerID="66618bf5f4000e93e6c657e7a3ffe6b8308f2a29aa99e02647ea636e195d07db" Oct 30 21:50:32.261448 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:50:32.261423 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66618bf5f4000e93e6c657e7a3ffe6b8308f2a29aa99e02647ea636e195d07db\": container with ID starting with 66618bf5f4000e93e6c657e7a3ffe6b8308f2a29aa99e02647ea636e195d07db not found: ID does not exist" containerID="66618bf5f4000e93e6c657e7a3ffe6b8308f2a29aa99e02647ea636e195d07db" Oct 30 21:50:32.261529 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.261459 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66618bf5f4000e93e6c657e7a3ffe6b8308f2a29aa99e02647ea636e195d07db"} err="failed to get container status \"66618bf5f4000e93e6c657e7a3ffe6b8308f2a29aa99e02647ea636e195d07db\": rpc error: code = NotFound desc = could not find container \"66618bf5f4000e93e6c657e7a3ffe6b8308f2a29aa99e02647ea636e195d07db\": container with ID starting with 66618bf5f4000e93e6c657e7a3ffe6b8308f2a29aa99e02647ea636e195d07db not found: ID does not exist" Oct 30 21:50:32.264375 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.264355 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl"] Oct 30 21:50:32.271188 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.271138 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f8144d-687488b9d7-n4qgl"] Oct 30 21:50:32.277312 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.277290 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j"] Oct 30 21:50:32.283264 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.283238 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f8144d-6dfb967f7c-s6h7j"] Oct 30 21:50:32.426931 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:32.426894 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:33.145842 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:33.145729 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:33.237547 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:33.237512 5801 generic.go:356] "Generic (PLEG): container finished" podID="239d3785-9a31-40ed-9e63-2d7e1265b5a2" containerID="6c3d90cae6558521bde9d8ddb74aec810038ed575c9b634e214e3158f58a9763" exitCode=0 Oct 30 21:50:33.237732 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:33.237646 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" event={"ID":"239d3785-9a31-40ed-9e63-2d7e1265b5a2","Type":"ContainerDied","Data":"6c3d90cae6558521bde9d8ddb74aec810038ed575c9b634e214e3158f58a9763"} Oct 30 21:50:33.238144 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:33.238095 5801 scope.go:122] "RemoveContainer" containerID="6c3d90cae6558521bde9d8ddb74aec810038ed575c9b634e214e3158f58a9763" Oct 30 21:50:33.239788 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:33.239725 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:50:33.239894 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:33.239808 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:50:33.239894 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:33.239861 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:50:33.449799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:33.449773 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm" Oct 30 21:50:33.525562 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:33.525487 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8be262d1-439b-4a2e-8658-e3ae067edc61" path="/var/lib/kubelet/pods/8be262d1-439b-4a2e-8658-e3ae067edc61/volumes" Oct 30 21:50:33.525956 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:33.525942 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d37caf73-c0d3-4433-8877-ee88415dd6c6" path="/var/lib/kubelet/pods/d37caf73-c0d3-4433-8877-ee88415dd6c6/volumes" Oct 30 21:50:33.543057 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:33.543029 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d28203f-5410-4c0e-b475-a0a4ab6afe74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d28203f-5410-4c0e-b475-a0a4ab6afe74-maas-api-xks-ca\") pod \"7d28203f-5410-4c0e-b475-a0a4ab6afe74\" (UID: \"7d28203f-5410-4c0e-b475-a0a4ab6afe74\") " Oct 30 21:50:33.543216 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:33.543165 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d28203f-5410-4c0e-b475-a0a4ab6afe74-kube-api-access-qvgx2\" (UniqueName: \"kubernetes.io/projected/7d28203f-5410-4c0e-b475-a0a4ab6afe74-kube-api-access-qvgx2\") pod \"7d28203f-5410-4c0e-b475-a0a4ab6afe74\" (UID: \"7d28203f-5410-4c0e-b475-a0a4ab6afe74\") " Oct 30 21:50:33.543286 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:33.543223 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d28203f-5410-4c0e-b475-a0a4ab6afe74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d28203f-5410-4c0e-b475-a0a4ab6afe74-maas-api-openshift-ca\") pod \"7d28203f-5410-4c0e-b475-a0a4ab6afe74\" (UID: \"7d28203f-5410-4c0e-b475-a0a4ab6afe74\") " Oct 30 21:50:33.543641 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:33.543615 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d28203f-5410-4c0e-b475-a0a4ab6afe74-maas-api-openshift-ca" pod "7d28203f-5410-4c0e-b475-a0a4ab6afe74" (UID: "7d28203f-5410-4c0e-b475-a0a4ab6afe74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:33.543744 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:33.543712 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d28203f-5410-4c0e-b475-a0a4ab6afe74-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:33.545431 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:33.545399 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d28203f-5410-4c0e-b475-a0a4ab6afe74-kube-api-access-qvgx2" pod "7d28203f-5410-4c0e-b475-a0a4ab6afe74" (UID: "7d28203f-5410-4c0e-b475-a0a4ab6afe74"). InnerVolumeSpecName "kube-api-access-qvgx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:33.545535 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:33.545430 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d28203f-5410-4c0e-b475-a0a4ab6afe74-maas-api-xks-ca" pod "7d28203f-5410-4c0e-b475-a0a4ab6afe74" (UID: "7d28203f-5410-4c0e-b475-a0a4ab6afe74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:33.645910 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:33.645877 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d28203f-5410-4c0e-b475-a0a4ab6afe74-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:33.646037 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:33.645914 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvgx2\" (UniqueName: \"kubernetes.io/projected/7d28203f-5410-4c0e-b475-a0a4ab6afe74-kube-api-access-qvgx2\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:33.937033 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:33.936989 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:34.207269 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:34.204546 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:34.250193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:34.250158 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm" event={"ID":"7d28203f-5410-4c0e-b475-a0a4ab6afe74","Type":"ContainerDied","Data":"1653f25a2a143dd29b77936cc6765f3f4165044d3663a57d08a8199b9636ec10"} Oct 30 21:50:34.250571 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:34.250216 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1653f25a2a143dd29b77936cc6765f3f4165044d3663a57d08a8199b9636ec10" Oct 30 21:50:34.250571 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:34.250291 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm" Oct 30 21:50:34.282939 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:34.282896 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" event={"ID":"239d3785-9a31-40ed-9e63-2d7e1265b5a2","Type":"ContainerStarted","Data":"271fffe223a2d2862e0fc018dc5681a54424333b8242d1a7a11ef7b22dcff634"} Oct 30 21:50:34.313549 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:34.313518 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:34.685955 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:34.685918 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:34.796914 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:34.796880 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:34.899903 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:34.899858 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:35.110784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.083420 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:35.136801 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.133566 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b1b1870-xwj5b"] Oct 30 21:50:35.228775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.202467 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:50:35.230244 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.229619 5801 patch_prober.go:29] interesting pod/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 30 21:50:35.230244 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.229674 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" podUID="239d3785-9a31-40ed-9e63-2d7e1265b5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 30 21:50:35.231016 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.230674 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b1b1870-xwj5b"] Oct 30 21:50:35.439847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.436122 5801 generic.go:356] "Generic (PLEG): container finished" podID="f9975b4c-127e-4130-ae64-b8785b0acc7f" containerID="6413883aa622617e7e9324d67928ea69d7313b63d4be87ca5628bf65627aabc8" exitCode=0 Oct 30 21:50:35.439847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.436228 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" event={"ID":"f9975b4c-127e-4130-ae64-b8785b0acc7f","Type":"ContainerDied","Data":"6413883aa622617e7e9324d67928ea69d7313b63d4be87ca5628bf65627aabc8"} Oct 30 21:50:35.439847 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.436286 5801 scope.go:122] "RemoveContainer" containerID="cb9ad72496a0cd9cbd1dd57911db06b323ed7484e3511285d9020bed8a079349" Oct 30 21:50:35.528883 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.528858 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2720213-8297-483a-8372-880429065ae5" path="/var/lib/kubelet/pods/b2720213-8297-483a-8372-880429065ae5/volumes" Oct 30 21:50:35.616585 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.616557 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx"] Oct 30 21:50:35.617176 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.617162 5801 cpu_manager.go:414] "containerMap: removing container" podUID="7d28203f-5410-4c0e-b475-a0a4ab6afe74" containerName="revoke-keys" Oct 30 21:50:35.617224 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.617178 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d28203f-5410-4c0e-b475-a0a4ab6afe74" containerName="revoke-keys" Oct 30 21:50:35.617224 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.617186 5801 cpu_manager.go:414] "containerMap: removing container" podUID="8be262d1-439b-4a2e-8658-e3ae067edc61" containerName="payload-pre-processing" Oct 30 21:50:35.617224 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.617191 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be262d1-439b-4a2e-8658-e3ae067edc61" containerName="payload-pre-processing" Oct 30 21:50:35.617224 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.617220 5801 cpu_manager.go:414] "containerMap: removing container" podUID="d37caf73-c0d3-4433-8877-ee88415dd6c6" containerName="payload-processing" Oct 30 21:50:35.617224 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.617225 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d37caf73-c0d3-4433-8877-ee88415dd6c6" containerName="payload-processing" Oct 30 21:50:35.617389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.617307 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="d37caf73-c0d3-4433-8877-ee88415dd6c6" containerName="payload-processing" Oct 30 21:50:35.617389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.617323 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d28203f-5410-4c0e-b475-a0a4ab6afe74" containerName="revoke-keys" Oct 30 21:50:35.617389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.617330 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be262d1-439b-4a2e-8658-e3ae067edc61" containerName="payload-pre-processing" Oct 30 21:50:35.622002 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.621984 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx" Oct 30 21:50:35.626335 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.626312 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:50:35.626530 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.626428 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:50:35.626677 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.626652 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:50:35.630314 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.630294 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx"] Oct 30 21:50:35.681129 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.681092 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:50:35.681303 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.681219 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:50:35.693815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.693022 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-workload-certs\") pod \"f9975b4c-127e-4130-ae64-b8785b0acc7f\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " Oct 30 21:50:35.693815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.693081 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-workload-socket\") pod \"f9975b4c-127e-4130-ae64-b8785b0acc7f\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " Oct 30 21:50:35.693815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.693152 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-credential-socket\") pod \"f9975b4c-127e-4130-ae64-b8785b0acc7f\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " Oct 30 21:50:35.693815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.693293 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9975b4c-127e-4130-ae64-b8785b0acc7f-kube-api-access-k9g8z\" (UniqueName: \"kubernetes.io/projected/f9975b4c-127e-4130-ae64-b8785b0acc7f-kube-api-access-k9g8z\") pod \"f9975b4c-127e-4130-ae64-b8785b0acc7f\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " Oct 30 21:50:35.693815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.693329 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-podinfo\") pod \"f9975b4c-127e-4130-ae64-b8785b0acc7f\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " Oct 30 21:50:35.693815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.693452 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9975b4c-127e-4130-ae64-b8785b0acc7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9975b4c-127e-4130-ae64-b8785b0acc7f-istiod-ca-cert\") pod \"f9975b4c-127e-4130-ae64-b8785b0acc7f\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " Oct 30 21:50:35.693815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.693492 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-data\") pod \"f9975b4c-127e-4130-ae64-b8785b0acc7f\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " Oct 30 21:50:35.693815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.693521 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-envoy\") pod \"f9975b4c-127e-4130-ae64-b8785b0acc7f\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " Oct 30 21:50:35.693815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.693587 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-token\" (UniqueName: \"kubernetes.io/projected/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-token\") pod \"f9975b4c-127e-4130-ae64-b8785b0acc7f\" (UID: \"f9975b4c-127e-4130-ae64-b8785b0acc7f\") " Oct 30 21:50:35.694183 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.693991 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d74e114c-82f3-4540-9f23-8bf7cbd317f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx\" (UID: \"d74e114c-82f3-4540-9f23-8bf7cbd317f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx" Oct 30 21:50:35.694183 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.694097 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d74e114c-82f3-4540-9f23-8bf7cbd317f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx\" (UID: \"d74e114c-82f3-4540-9f23-8bf7cbd317f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx" Oct 30 21:50:35.694493 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.694469 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-workload-certs" pod "f9975b4c-127e-4130-ae64-b8785b0acc7f" (UID: "f9975b4c-127e-4130-ae64-b8785b0acc7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:35.694722 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.694701 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-workload-socket" pod "f9975b4c-127e-4130-ae64-b8785b0acc7f" (UID: "f9975b4c-127e-4130-ae64-b8785b0acc7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:35.694927 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.694906 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-credential-socket" pod "f9975b4c-127e-4130-ae64-b8785b0acc7f" (UID: "f9975b4c-127e-4130-ae64-b8785b0acc7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:35.698058 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.698029 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6z52\" (UniqueName: \"kubernetes.io/projected/d74e114c-82f3-4540-9f23-8bf7cbd317f3-kube-api-access-p6z52\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx\" (UID: \"d74e114c-82f3-4540-9f23-8bf7cbd317f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx" Oct 30 21:50:35.698156 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.698125 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:35.698200 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.698145 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:35.700073 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.700047 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-data" pod "f9975b4c-127e-4130-ae64-b8785b0acc7f" (UID: "f9975b4c-127e-4130-ae64-b8785b0acc7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:35.701276 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.701240 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9975b4c-127e-4130-ae64-b8785b0acc7f-kube-api-access-k9g8z" pod "f9975b4c-127e-4130-ae64-b8785b0acc7f" (UID: "f9975b4c-127e-4130-ae64-b8785b0acc7f"). InnerVolumeSpecName "kube-api-access-k9g8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:35.701889 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.701865 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9975b4c-127e-4130-ae64-b8785b0acc7f-istiod-ca-cert" pod "f9975b4c-127e-4130-ae64-b8785b0acc7f" (UID: "f9975b4c-127e-4130-ae64-b8785b0acc7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:35.703117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.703093 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-podinfo" pod "f9975b4c-127e-4130-ae64-b8785b0acc7f" (UID: "f9975b4c-127e-4130-ae64-b8785b0acc7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:35.703117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.703107 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-token" pod "f9975b4c-127e-4130-ae64-b8785b0acc7f" (UID: "f9975b4c-127e-4130-ae64-b8785b0acc7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:35.703281 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.703194 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-envoy" pod "f9975b4c-127e-4130-ae64-b8785b0acc7f" (UID: "f9975b4c-127e-4130-ae64-b8785b0acc7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:35.798745 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.798702 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d74e114c-82f3-4540-9f23-8bf7cbd317f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx\" (UID: \"d74e114c-82f3-4540-9f23-8bf7cbd317f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx" Oct 30 21:50:35.798976 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.798880 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6z52\" (UniqueName: \"kubernetes.io/projected/d74e114c-82f3-4540-9f23-8bf7cbd317f3-kube-api-access-p6z52\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx\" (UID: \"d74e114c-82f3-4540-9f23-8bf7cbd317f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx" Oct 30 21:50:35.799057 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.799010 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d74e114c-82f3-4540-9f23-8bf7cbd317f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx\" (UID: \"d74e114c-82f3-4540-9f23-8bf7cbd317f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx" Oct 30 21:50:35.799126 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.799113 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9g8z\" (UniqueName: \"kubernetes.io/projected/f9975b4c-127e-4130-ae64-b8785b0acc7f-kube-api-access-k9g8z\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:35.799185 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.799134 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:35.799185 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.799151 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9975b4c-127e-4130-ae64-b8785b0acc7f-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:35.799185 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.799165 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:35.799316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.799191 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:35.799316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.799206 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9975b4c-127e-4130-ae64-b8785b0acc7f-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:35.799316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.799222 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9975b4c-127e-4130-ae64-b8785b0acc7f-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:35.799411 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.799346 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d74e114c-82f3-4540-9f23-8bf7cbd317f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx\" (UID: \"d74e114c-82f3-4540-9f23-8bf7cbd317f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx" Oct 30 21:50:35.801241 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.801220 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d74e114c-82f3-4540-9f23-8bf7cbd317f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx\" (UID: \"d74e114c-82f3-4540-9f23-8bf7cbd317f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx" Oct 30 21:50:35.811481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.811455 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6z52\" (UniqueName: \"kubernetes.io/projected/d74e114c-82f3-4540-9f23-8bf7cbd317f3-kube-api-access-p6z52\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx\" (UID: \"d74e114c-82f3-4540-9f23-8bf7cbd317f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx" Oct 30 21:50:35.938683 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:35.938651 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx" Oct 30 21:50:36.018874 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:36.018845 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:36.199859 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:36.199813 5801 patch_prober.go:29] interesting pod/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 30 21:50:36.200035 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:36.199896 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" podUID="239d3785-9a31-40ed-9e63-2d7e1265b5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 30 21:50:36.283261 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:36.283229 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx"] Oct 30 21:50:36.283860 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:50:36.283833 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74e114c_82f3_4540_9f23_8bf7cbd317f3.slice/crio-61cf8f688b6ea97aab1d68d770e12913f9a18c605136e407dc6f03135f93eb8b WatchSource:0}: Error finding container 61cf8f688b6ea97aab1d68d770e12913f9a18c605136e407dc6f03135f93eb8b: Status 404 returned error can't find the container with id 61cf8f688b6ea97aab1d68d770e12913f9a18c605136e407dc6f03135f93eb8b Oct 30 21:50:36.446855 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:36.446821 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx" event={"ID":"d74e114c-82f3-4540-9f23-8bf7cbd317f3","Type":"ContainerStarted","Data":"61cf8f688b6ea97aab1d68d770e12913f9a18c605136e407dc6f03135f93eb8b"} Oct 30 21:50:36.448296 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:36.448271 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" Oct 30 21:50:36.448456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:36.448299 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5" event={"ID":"f9975b4c-127e-4130-ae64-b8785b0acc7f","Type":"ContainerDied","Data":"1517e292fd5667b2c5882e433e5b4f5e78c0f2c0d1c9fdfd5383ceb2afa9d890"} Oct 30 21:50:36.448456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:36.448348 5801 scope.go:122] "RemoveContainer" containerID="6413883aa622617e7e9324d67928ea69d7313b63d4be87ca5628bf65627aabc8" Oct 30 21:50:36.509786 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:36.509743 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5"] Oct 30 21:50:36.517403 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:36.517375 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-950784-openshift-default-b4f5cd5fc-d7hh5"] Oct 30 21:50:36.835882 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:36.835829 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:37.200402 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:37.200367 5801 patch_prober.go:29] interesting pod/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 30 21:50:37.200577 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:37.200420 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" podUID="239d3785-9a31-40ed-9e63-2d7e1265b5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 30 21:50:37.481971 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:37.481885 5801 generic.go:356] "Generic (PLEG): container finished" podID="d74e114c-82f3-4540-9f23-8bf7cbd317f3" containerID="50865f66f989b811e6056c622767da272e9f76d5495fb27b0be3686708ac8d9f" exitCode=0 Oct 30 21:50:37.482396 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:37.481967 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx" event={"ID":"d74e114c-82f3-4540-9f23-8bf7cbd317f3","Type":"ContainerDied","Data":"50865f66f989b811e6056c622767da272e9f76d5495fb27b0be3686708ac8d9f"} Oct 30 21:50:37.525390 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:37.525357 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9975b4c-127e-4130-ae64-b8785b0acc7f" path="/var/lib/kubelet/pods/f9975b4c-127e-4130-ae64-b8785b0acc7f/volumes" Oct 30 21:50:37.606310 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:37.606273 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:38.199932 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:38.199902 5801 patch_prober.go:29] interesting pod/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 30 21:50:38.200100 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:38.199978 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" podUID="239d3785-9a31-40ed-9e63-2d7e1265b5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 30 21:50:38.242172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:38.242148 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:50:38.278823 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:38.278780 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:38.690137 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:38.690111 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx" Oct 30 21:50:38.742078 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:38.742047 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d74e114c-82f3-4540-9f23-8bf7cbd317f3-kube-api-access-p6z52\" (UniqueName: \"kubernetes.io/projected/d74e114c-82f3-4540-9f23-8bf7cbd317f3-kube-api-access-p6z52\") pod \"d74e114c-82f3-4540-9f23-8bf7cbd317f3\" (UID: \"d74e114c-82f3-4540-9f23-8bf7cbd317f3\") " Oct 30 21:50:38.742343 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:38.742322 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d74e114c-82f3-4540-9f23-8bf7cbd317f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d74e114c-82f3-4540-9f23-8bf7cbd317f3-maas-api-openshift-ca\") pod \"d74e114c-82f3-4540-9f23-8bf7cbd317f3\" (UID: \"d74e114c-82f3-4540-9f23-8bf7cbd317f3\") " Oct 30 21:50:38.742445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:38.742422 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d74e114c-82f3-4540-9f23-8bf7cbd317f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d74e114c-82f3-4540-9f23-8bf7cbd317f3-maas-api-xks-ca\") pod \"d74e114c-82f3-4540-9f23-8bf7cbd317f3\" (UID: \"d74e114c-82f3-4540-9f23-8bf7cbd317f3\") " Oct 30 21:50:38.742790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:38.742725 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74e114c-82f3-4540-9f23-8bf7cbd317f3-maas-api-openshift-ca" pod "d74e114c-82f3-4540-9f23-8bf7cbd317f3" (UID: "d74e114c-82f3-4540-9f23-8bf7cbd317f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:38.743029 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:38.742980 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d74e114c-82f3-4540-9f23-8bf7cbd317f3-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:38.744419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:38.744376 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74e114c-82f3-4540-9f23-8bf7cbd317f3-kube-api-access-p6z52" pod "d74e114c-82f3-4540-9f23-8bf7cbd317f3" (UID: "d74e114c-82f3-4540-9f23-8bf7cbd317f3"). InnerVolumeSpecName "kube-api-access-p6z52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:38.744419 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:38.744385 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d74e114c-82f3-4540-9f23-8bf7cbd317f3-maas-api-xks-ca" pod "d74e114c-82f3-4540-9f23-8bf7cbd317f3" (UID: "d74e114c-82f3-4540-9f23-8bf7cbd317f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:38.837880 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:38.837841 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:38.844998 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:38.844975 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d74e114c-82f3-4540-9f23-8bf7cbd317f3-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:38.844998 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:38.844999 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6z52\" (UniqueName: \"kubernetes.io/projected/d74e114c-82f3-4540-9f23-8bf7cbd317f3-kube-api-access-p6z52\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:38.979292 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:38.979255 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:38.998996 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:38.998925 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz"] Oct 30 21:50:39.002616 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:39.002593 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8144d-bqfkz"] Oct 30 21:50:39.025337 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:39.025306 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e02f8610-g9grl"] Oct 30 21:50:39.029221 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:39.029183 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e02f8610-g9grl"] Oct 30 21:50:39.083655 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:39.083615 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:39.198673 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:39.198639 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:39.199914 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:39.199885 5801 patch_prober.go:29] interesting pod/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 30 21:50:39.200107 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:39.200085 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" podUID="239d3785-9a31-40ed-9e63-2d7e1265b5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 30 21:50:39.504810 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:39.504742 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx" Oct 30 21:50:39.504810 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:39.504791 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx" event={"ID":"d74e114c-82f3-4540-9f23-8bf7cbd317f3","Type":"ContainerDied","Data":"61cf8f688b6ea97aab1d68d770e12913f9a18c605136e407dc6f03135f93eb8b"} Oct 30 21:50:39.505030 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:39.504860 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61cf8f688b6ea97aab1d68d770e12913f9a18c605136e407dc6f03135f93eb8b" Oct 30 21:50:39.526637 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:39.526597 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f2c9cba-2588-4e81-9a7c-33fef0ffb58e" path="/var/lib/kubelet/pods/2f2c9cba-2588-4e81-9a7c-33fef0ffb58e/volumes" Oct 30 21:50:39.527190 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:39.527166 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db889eca-5e32-4528-b653-c72e065bd1e5" path="/var/lib/kubelet/pods/db889eca-5e32-4528-b653-c72e065bd1e5/volumes" Oct 30 21:50:39.538725 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:39.538425 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:39.944468 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:39.944427 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:40.204652 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:40.204576 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:50:40.205669 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:40.205650 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:50:40.730457 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:40.730419 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5"] Oct 30 21:50:40.755399 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:40.755361 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:40.858339 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:40.858305 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:41.639238 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:41.638214 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:41.741193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:41.741158 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:41.766474 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:41.766443 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9"] Oct 30 21:50:41.770080 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:41.770052 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2b36c1-g6td9"] Oct 30 21:50:41.846099 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:41.846065 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:42.084125 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:42.084081 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:42.165973 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:42.165930 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm"] Oct 30 21:50:42.169915 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:42.169888 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6cf5f0-mztpm"] Oct 30 21:50:42.189450 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:42.189418 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:42.291313 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:42.291276 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:42.393110 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:42.391520 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:42.489947 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:42.489906 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:42.703277 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:42.703167 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:42.947595 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:42.947558 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:43.525678 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:43.525642 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3658bb0a-6bcf-47a4-a1f9-fff73c104c17" path="/var/lib/kubelet/pods/3658bb0a-6bcf-47a4-a1f9-fff73c104c17/volumes" Oct 30 21:50:43.526197 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:43.526181 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d28203f-5410-4c0e-b475-a0a4ab6afe74" path="/var/lib/kubelet/pods/7d28203f-5410-4c0e-b475-a0a4ab6afe74/volumes" Oct 30 21:50:43.636476 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:43.636434 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:44.444085 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:44.444049 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:45.238822 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:45.238789 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:45.731002 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:45.730958 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5" podUID="c6a121f3-14a8-447c-bdad-ac188d32830c" containerName="payload-pre-processing" containerID="cri-o://e8ee73657ede5796664e11bcaa0e746d993c0a931a5aed027108dab27394a2cd" gracePeriod=25 Oct 30 21:50:45.749332 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:45.749305 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx"] Oct 30 21:50:45.767851 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:45.767803 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw"] Oct 30 21:50:45.828678 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:45.828638 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:45.927550 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:45.927472 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:46.009251 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:46.009223 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5" Oct 30 21:50:46.085936 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:46.085893 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6a121f3-14a8-447c-bdad-ac188d32830c-kube-api-access-8gzqp\" (UniqueName: \"kubernetes.io/projected/c6a121f3-14a8-447c-bdad-ac188d32830c-kube-api-access-8gzqp\") pod \"c6a121f3-14a8-447c-bdad-ac188d32830c\" (UID: \"c6a121f3-14a8-447c-bdad-ac188d32830c\") " Oct 30 21:50:46.086180 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:46.086156 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6a121f3-14a8-447c-bdad-ac188d32830c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a121f3-14a8-447c-bdad-ac188d32830c-plugins-config-volume\") pod \"c6a121f3-14a8-447c-bdad-ac188d32830c\" (UID: \"c6a121f3-14a8-447c-bdad-ac188d32830c\") " Oct 30 21:50:46.086626 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:46.086590 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a121f3-14a8-447c-bdad-ac188d32830c-plugins-config-volume" pod "c6a121f3-14a8-447c-bdad-ac188d32830c" (UID: "c6a121f3-14a8-447c-bdad-ac188d32830c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:46.086994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:46.086975 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a121f3-14a8-447c-bdad-ac188d32830c-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:46.088815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:46.088744 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a121f3-14a8-447c-bdad-ac188d32830c-kube-api-access-8gzqp" pod "c6a121f3-14a8-447c-bdad-ac188d32830c" (UID: "c6a121f3-14a8-447c-bdad-ac188d32830c"). InnerVolumeSpecName "kube-api-access-8gzqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:46.154776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:46.151453 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:46.203848 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:46.203813 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gzqp\" (UniqueName: \"kubernetes.io/projected/c6a121f3-14a8-447c-bdad-ac188d32830c-kube-api-access-8gzqp\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:46.580979 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:46.580944 5801 generic.go:356] "Generic (PLEG): container finished" podID="c6a121f3-14a8-447c-bdad-ac188d32830c" containerID="e8ee73657ede5796664e11bcaa0e746d993c0a931a5aed027108dab27394a2cd" exitCode=0 Oct 30 21:50:46.581165 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:46.580987 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5" Oct 30 21:50:46.581165 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:46.581008 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5" event={"ID":"c6a121f3-14a8-447c-bdad-ac188d32830c","Type":"ContainerDied","Data":"e8ee73657ede5796664e11bcaa0e746d993c0a931a5aed027108dab27394a2cd"} Oct 30 21:50:46.581165 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:46.581063 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5" event={"ID":"c6a121f3-14a8-447c-bdad-ac188d32830c","Type":"ContainerDied","Data":"aec8c7472c9ee95788f795c8681b86c5c634fe49bda4dd16cb1875154f91a98b"} Oct 30 21:50:46.581165 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:46.581091 5801 scope.go:122] "RemoveContainer" containerID="e8ee73657ede5796664e11bcaa0e746d993c0a931a5aed027108dab27394a2cd" Oct 30 21:50:46.604001 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:46.603978 5801 scope.go:122] "RemoveContainer" containerID="e8ee73657ede5796664e11bcaa0e746d993c0a931a5aed027108dab27394a2cd" Oct 30 21:50:46.604358 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:50:46.604334 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8ee73657ede5796664e11bcaa0e746d993c0a931a5aed027108dab27394a2cd\": container with ID starting with e8ee73657ede5796664e11bcaa0e746d993c0a931a5aed027108dab27394a2cd not found: ID does not exist" containerID="e8ee73657ede5796664e11bcaa0e746d993c0a931a5aed027108dab27394a2cd" Oct 30 21:50:46.604415 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:46.604366 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ee73657ede5796664e11bcaa0e746d993c0a931a5aed027108dab27394a2cd"} err="failed to get container status \"e8ee73657ede5796664e11bcaa0e746d993c0a931a5aed027108dab27394a2cd\": rpc error: code = NotFound desc = could not find container \"e8ee73657ede5796664e11bcaa0e746d993c0a931a5aed027108dab27394a2cd\": container with ID starting with e8ee73657ede5796664e11bcaa0e746d993c0a931a5aed027108dab27394a2cd not found: ID does not exist" Oct 30 21:50:46.609050 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:46.609015 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5"] Oct 30 21:50:46.615924 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:46.615894 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6cf5f0-5cb66474bc-9pjt5"] Oct 30 21:50:46.777777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:46.776585 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:47.428115 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:47.428072 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:47.532777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:47.528718 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:47.545731 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:47.545698 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6a121f3-14a8-447c-bdad-ac188d32830c" path="/var/lib/kubelet/pods/c6a121f3-14a8-447c-bdad-ac188d32830c/volumes" Oct 30 21:50:47.789265 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:47.789179 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf"] Oct 30 21:50:47.789936 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:47.789894 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" podUID="0e91a97c-e981-4ab6-b6cd-3f92381ed5c4" containerName="istio-proxy" containerID="cri-o://6200424ff91b7331861a8103fd76e92a79a228b40526b3f653f80e377e873dcf" gracePeriod=30 Oct 30 21:50:47.876692 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:47.876589 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:48.567778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.567633 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg"] Oct 30 21:50:48.568784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.568572 5801 cpu_manager.go:414] "containerMap: removing container" podUID="d74e114c-82f3-4540-9f23-8bf7cbd317f3" containerName="revoke-keys" Oct 30 21:50:48.568784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.568595 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d74e114c-82f3-4540-9f23-8bf7cbd317f3" containerName="revoke-keys" Oct 30 21:50:48.568784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.568632 5801 cpu_manager.go:414] "containerMap: removing container" podUID="c6a121f3-14a8-447c-bdad-ac188d32830c" containerName="payload-pre-processing" Oct 30 21:50:48.568784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.568641 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6a121f3-14a8-447c-bdad-ac188d32830c" containerName="payload-pre-processing" Oct 30 21:50:48.568784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.568662 5801 cpu_manager.go:414] "containerMap: removing container" podUID="f9975b4c-127e-4130-ae64-b8785b0acc7f" containerName="istio-proxy" Oct 30 21:50:48.568784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.568670 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9975b4c-127e-4130-ae64-b8785b0acc7f" containerName="istio-proxy" Oct 30 21:50:48.568784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.568687 5801 cpu_manager.go:414] "containerMap: removing container" podUID="f9975b4c-127e-4130-ae64-b8785b0acc7f" containerName="istio-proxy" Oct 30 21:50:48.568784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.568694 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9975b4c-127e-4130-ae64-b8785b0acc7f" containerName="istio-proxy" Oct 30 21:50:48.569321 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.568969 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6a121f3-14a8-447c-bdad-ac188d32830c" containerName="payload-pre-processing" Oct 30 21:50:48.569321 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.568990 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9975b4c-127e-4130-ae64-b8785b0acc7f" containerName="istio-proxy" Oct 30 21:50:48.569321 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.569003 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9975b4c-127e-4130-ae64-b8785b0acc7f" containerName="istio-proxy" Oct 30 21:50:48.569321 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.569024 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="d74e114c-82f3-4540-9f23-8bf7cbd317f3" containerName="revoke-keys" Oct 30 21:50:48.577362 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.577337 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg" Oct 30 21:50:48.584539 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.584508 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg"] Oct 30 21:50:48.614661 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.609881 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:50:48.614857 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.614739 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:50:48.638650 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.638620 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:50:48.638877 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.638813 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:50:48.678735 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.678693 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d5666d-c9d2-4570-a859-82622af82893-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg\" (UID: \"e9d5666d-c9d2-4570-a859-82622af82893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg" Oct 30 21:50:48.679212 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.679189 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d5666d-c9d2-4570-a859-82622af82893-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg\" (UID: \"e9d5666d-c9d2-4570-a859-82622af82893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg" Oct 30 21:50:48.679345 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.679325 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml29c\" (UniqueName: \"kubernetes.io/projected/e9d5666d-c9d2-4570-a859-82622af82893-kube-api-access-ml29c\") pod \"maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg\" (UID: \"e9d5666d-c9d2-4570-a859-82622af82893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg" Oct 30 21:50:48.687673 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.687637 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:48.784164 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.783206 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d5666d-c9d2-4570-a859-82622af82893-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg\" (UID: \"e9d5666d-c9d2-4570-a859-82622af82893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg" Oct 30 21:50:48.784164 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.783285 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml29c\" (UniqueName: \"kubernetes.io/projected/e9d5666d-c9d2-4570-a859-82622af82893-kube-api-access-ml29c\") pod \"maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg\" (UID: \"e9d5666d-c9d2-4570-a859-82622af82893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg" Oct 30 21:50:48.784164 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.783439 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d5666d-c9d2-4570-a859-82622af82893-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg\" (UID: \"e9d5666d-c9d2-4570-a859-82622af82893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg" Oct 30 21:50:48.784439 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.784186 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d5666d-c9d2-4570-a859-82622af82893-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg\" (UID: \"e9d5666d-c9d2-4570-a859-82622af82893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg" Oct 30 21:50:48.787823 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.787793 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d5666d-c9d2-4570-a859-82622af82893-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg\" (UID: \"e9d5666d-c9d2-4570-a859-82622af82893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg" Oct 30 21:50:48.798159 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.797990 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml29c\" (UniqueName: \"kubernetes.io/projected/e9d5666d-c9d2-4570-a859-82622af82893-kube-api-access-ml29c\") pod \"maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg\" (UID: \"e9d5666d-c9d2-4570-a859-82622af82893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg" Oct 30 21:50:48.908181 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:48.908097 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg" Oct 30 21:50:49.217559 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:49.217524 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg"] Oct 30 21:50:49.219268 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:50:49.219241 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d5666d_c9d2_4570_a859_82622af82893.slice/crio-4e900a0bcd2eba1d8d78490953003d43d4e0fec8aa317d0e110d945b494e9709 WatchSource:0}: Error finding container 4e900a0bcd2eba1d8d78490953003d43d4e0fec8aa317d0e110d945b494e9709: Status 404 returned error can't find the container with id 4e900a0bcd2eba1d8d78490953003d43d4e0fec8aa317d0e110d945b494e9709 Oct 30 21:50:49.415267 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:49.415229 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:49.631063 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:49.631028 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg" event={"ID":"e9d5666d-c9d2-4570-a859-82622af82893","Type":"ContainerStarted","Data":"4e900a0bcd2eba1d8d78490953003d43d4e0fec8aa317d0e110d945b494e9709"} Oct 30 21:50:50.127667 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:50.127629 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:50.489866 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:50.489808 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:50.595598 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:50.595546 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:50.640534 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:50.640444 5801 generic.go:356] "Generic (PLEG): container finished" podID="e9d5666d-c9d2-4570-a859-82622af82893" containerID="8a7d8ee01470f3fc7d45bcb864f4f4b7e64771f1d52b26ebe4d8f7f511e7d8ee" exitCode=0 Oct 30 21:50:50.640699 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:50.640540 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg" event={"ID":"e9d5666d-c9d2-4570-a859-82622af82893","Type":"ContainerDied","Data":"8a7d8ee01470f3fc7d45bcb864f4f4b7e64771f1d52b26ebe4d8f7f511e7d8ee"} Oct 30 21:50:50.750337 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:50.750291 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx" podUID="cb686228-0c5b-4d64-82b4-1838e0f75369" containerName="payload-processing" containerID="cri-o://e9f60dbe44c7866b366e1f6c1904473a4a30b1d870af01a0b4859a15071fb986" gracePeriod=25 Oct 30 21:50:50.778121 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:50.777911 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw" podUID="a5559491-62e9-4637-9253-ac2edfccbd46" containerName="payload-pre-processing" containerID="cri-o://2d7209188df30f2692627a27618727640e7899b327132ac72e6221645bc6c72a" gracePeriod=25 Oct 30 21:50:50.836794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:50.835156 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5"] Oct 30 21:50:50.836794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:50.835415 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" podUID="a7e5545a-7bc6-437b-bef6-4c6ee6776f09" containerName="istio-proxy" containerID="cri-o://1e072289193d11c9d353f41dabe560444a8737a005c53be22a79c809d6764f16" gracePeriod=30 Oct 30 21:50:50.839903 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:50.839853 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:50.980780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:50.978318 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:51.196306 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.196159 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:51.254060 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.254031 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx" Oct 30 21:50:51.294664 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.294624 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb686228-0c5b-4d64-82b4-1838e0f75369-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb686228-0c5b-4d64-82b4-1838e0f75369-plugins-config-volume\") pod \"cb686228-0c5b-4d64-82b4-1838e0f75369\" (UID: \"cb686228-0c5b-4d64-82b4-1838e0f75369\") " Oct 30 21:50:51.294891 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.294870 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb686228-0c5b-4d64-82b4-1838e0f75369-kube-api-access-p97xk\" (UniqueName: \"kubernetes.io/projected/cb686228-0c5b-4d64-82b4-1838e0f75369-kube-api-access-p97xk\") pod \"cb686228-0c5b-4d64-82b4-1838e0f75369\" (UID: \"cb686228-0c5b-4d64-82b4-1838e0f75369\") " Oct 30 21:50:51.295350 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.295324 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb686228-0c5b-4d64-82b4-1838e0f75369-plugins-config-volume" pod "cb686228-0c5b-4d64-82b4-1838e0f75369" (UID: "cb686228-0c5b-4d64-82b4-1838e0f75369"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:51.309007 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.308964 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb686228-0c5b-4d64-82b4-1838e0f75369-kube-api-access-p97xk" pod "cb686228-0c5b-4d64-82b4-1838e0f75369" (UID: "cb686228-0c5b-4d64-82b4-1838e0f75369"). InnerVolumeSpecName "kube-api-access-p97xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:51.389475 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.389330 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:51.397775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.397578 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p97xk\" (UniqueName: \"kubernetes.io/projected/cb686228-0c5b-4d64-82b4-1838e0f75369-kube-api-access-p97xk\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:51.397775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.397613 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb686228-0c5b-4d64-82b4-1838e0f75369-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:51.410383 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.408423 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7"] Oct 30 21:50:51.410383 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.409219 5801 cpu_manager.go:414] "containerMap: removing container" podUID="cb686228-0c5b-4d64-82b4-1838e0f75369" containerName="payload-processing" Oct 30 21:50:51.410383 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.409235 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb686228-0c5b-4d64-82b4-1838e0f75369" containerName="payload-processing" Oct 30 21:50:51.410383 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.409368 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb686228-0c5b-4d64-82b4-1838e0f75369" containerName="payload-processing" Oct 30 21:50:51.417228 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.414315 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7" Oct 30 21:50:51.508917 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.502831 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7"] Oct 30 21:50:51.617349 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.613726 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983f1693-9287-4523-a83f-70da14c30336-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7\" (UID: \"983f1693-9287-4523-a83f-70da14c30336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7" Oct 30 21:50:51.617349 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.613824 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983f1693-9287-4523-a83f-70da14c30336-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7\" (UID: \"983f1693-9287-4523-a83f-70da14c30336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7" Oct 30 21:50:51.617349 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.614022 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spggj\" (UniqueName: \"kubernetes.io/projected/983f1693-9287-4523-a83f-70da14c30336-kube-api-access-spggj\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7\" (UID: \"983f1693-9287-4523-a83f-70da14c30336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7" Oct 30 21:50:51.622855 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.622513 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw" Oct 30 21:50:51.650883 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.650856 5801 generic.go:356] "Generic (PLEG): container finished" podID="a5559491-62e9-4637-9253-ac2edfccbd46" containerID="2d7209188df30f2692627a27618727640e7899b327132ac72e6221645bc6c72a" exitCode=0 Oct 30 21:50:51.651182 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.651165 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw" Oct 30 21:50:51.651538 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.651513 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw" event={"ID":"a5559491-62e9-4637-9253-ac2edfccbd46","Type":"ContainerDied","Data":"2d7209188df30f2692627a27618727640e7899b327132ac72e6221645bc6c72a"} Oct 30 21:50:51.651709 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.651690 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw" event={"ID":"a5559491-62e9-4637-9253-ac2edfccbd46","Type":"ContainerDied","Data":"ce9b64a19201d1fc6aa341617ab1367af68faa814c0b98ccec8a167002156235"} Oct 30 21:50:51.651870 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.651853 5801 scope.go:122] "RemoveContainer" containerID="2d7209188df30f2692627a27618727640e7899b327132ac72e6221645bc6c72a" Oct 30 21:50:51.665931 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.665261 5801 generic.go:356] "Generic (PLEG): container finished" podID="cb686228-0c5b-4d64-82b4-1838e0f75369" containerID="e9f60dbe44c7866b366e1f6c1904473a4a30b1d870af01a0b4859a15071fb986" exitCode=0 Oct 30 21:50:51.665931 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.665692 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx" Oct 30 21:50:51.676797 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.666512 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx" event={"ID":"cb686228-0c5b-4d64-82b4-1838e0f75369","Type":"ContainerDied","Data":"e9f60dbe44c7866b366e1f6c1904473a4a30b1d870af01a0b4859a15071fb986"} Oct 30 21:50:51.676797 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.666588 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx" event={"ID":"cb686228-0c5b-4d64-82b4-1838e0f75369","Type":"ContainerDied","Data":"1b07fbdab8e7cec271ab5e7faa0d469f48410299527edb1b1ac15110174b6a94"} Oct 30 21:50:51.698748 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.698689 5801 scope.go:122] "RemoveContainer" containerID="2d7209188df30f2692627a27618727640e7899b327132ac72e6221645bc6c72a" Oct 30 21:50:51.699057 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:50:51.699029 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d7209188df30f2692627a27618727640e7899b327132ac72e6221645bc6c72a\": container with ID starting with 2d7209188df30f2692627a27618727640e7899b327132ac72e6221645bc6c72a not found: ID does not exist" containerID="2d7209188df30f2692627a27618727640e7899b327132ac72e6221645bc6c72a" Oct 30 21:50:51.699127 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.699070 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d7209188df30f2692627a27618727640e7899b327132ac72e6221645bc6c72a"} err="failed to get container status \"2d7209188df30f2692627a27618727640e7899b327132ac72e6221645bc6c72a\": rpc error: code = NotFound desc = could not find container \"2d7209188df30f2692627a27618727640e7899b327132ac72e6221645bc6c72a\": container with ID starting with 2d7209188df30f2692627a27618727640e7899b327132ac72e6221645bc6c72a not found: ID does not exist" Oct 30 21:50:51.699127 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.699092 5801 scope.go:122] "RemoveContainer" containerID="e9f60dbe44c7866b366e1f6c1904473a4a30b1d870af01a0b4859a15071fb986" Oct 30 21:50:51.712286 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.712257 5801 scope.go:122] "RemoveContainer" containerID="e9f60dbe44c7866b366e1f6c1904473a4a30b1d870af01a0b4859a15071fb986" Oct 30 21:50:51.712651 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:50:51.712627 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9f60dbe44c7866b366e1f6c1904473a4a30b1d870af01a0b4859a15071fb986\": container with ID starting with e9f60dbe44c7866b366e1f6c1904473a4a30b1d870af01a0b4859a15071fb986 not found: ID does not exist" containerID="e9f60dbe44c7866b366e1f6c1904473a4a30b1d870af01a0b4859a15071fb986" Oct 30 21:50:51.712731 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.712664 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9f60dbe44c7866b366e1f6c1904473a4a30b1d870af01a0b4859a15071fb986"} err="failed to get container status \"e9f60dbe44c7866b366e1f6c1904473a4a30b1d870af01a0b4859a15071fb986\": rpc error: code = NotFound desc = could not find container \"e9f60dbe44c7866b366e1f6c1904473a4a30b1d870af01a0b4859a15071fb986\": container with ID starting with e9f60dbe44c7866b366e1f6c1904473a4a30b1d870af01a0b4859a15071fb986 not found: ID does not exist" Oct 30 21:50:51.713659 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.713635 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx"] Oct 30 21:50:51.715415 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.714974 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5559491-62e9-4637-9253-ac2edfccbd46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5559491-62e9-4637-9253-ac2edfccbd46-plugins-config-volume\") pod \"a5559491-62e9-4637-9253-ac2edfccbd46\" (UID: \"a5559491-62e9-4637-9253-ac2edfccbd46\") " Oct 30 21:50:51.715415 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.715367 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5559491-62e9-4637-9253-ac2edfccbd46-plugins-config-volume" pod "a5559491-62e9-4637-9253-ac2edfccbd46" (UID: "a5559491-62e9-4637-9253-ac2edfccbd46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:51.716206 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.715250 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5559491-62e9-4637-9253-ac2edfccbd46-kube-api-access-b9fqf\" (UniqueName: \"kubernetes.io/projected/a5559491-62e9-4637-9253-ac2edfccbd46-kube-api-access-b9fqf\") pod \"a5559491-62e9-4637-9253-ac2edfccbd46\" (UID: \"a5559491-62e9-4637-9253-ac2edfccbd46\") " Oct 30 21:50:51.716206 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.715931 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983f1693-9287-4523-a83f-70da14c30336-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7\" (UID: \"983f1693-9287-4523-a83f-70da14c30336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7" Oct 30 21:50:51.716206 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.715962 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983f1693-9287-4523-a83f-70da14c30336-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7\" (UID: \"983f1693-9287-4523-a83f-70da14c30336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7" Oct 30 21:50:51.716206 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.716086 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spggj\" (UniqueName: \"kubernetes.io/projected/983f1693-9287-4523-a83f-70da14c30336-kube-api-access-spggj\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7\" (UID: \"983f1693-9287-4523-a83f-70da14c30336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7" Oct 30 21:50:51.716653 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.716237 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5559491-62e9-4637-9253-ac2edfccbd46-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:51.717634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.717611 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5559491-62e9-4637-9253-ac2edfccbd46-kube-api-access-b9fqf" pod "a5559491-62e9-4637-9253-ac2edfccbd46" (UID: "a5559491-62e9-4637-9253-ac2edfccbd46"). InnerVolumeSpecName "kube-api-access-b9fqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:51.718892 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.718866 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983f1693-9287-4523-a83f-70da14c30336-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7\" (UID: \"983f1693-9287-4523-a83f-70da14c30336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7" Oct 30 21:50:51.719598 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.719576 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983f1693-9287-4523-a83f-70da14c30336-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7\" (UID: \"983f1693-9287-4523-a83f-70da14c30336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7" Oct 30 21:50:51.725794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.725191 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fbf428-5d484c9bff-zqkgx"] Oct 30 21:50:51.729594 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.729570 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spggj\" (UniqueName: \"kubernetes.io/projected/983f1693-9287-4523-a83f-70da14c30336-kube-api-access-spggj\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7\" (UID: \"983f1693-9287-4523-a83f-70da14c30336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7" Oct 30 21:50:51.807719 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.807679 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:51.818318 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.818284 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9fqf\" (UniqueName: \"kubernetes.io/projected/a5559491-62e9-4637-9253-ac2edfccbd46-kube-api-access-b9fqf\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:51.831956 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.831925 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt"] Oct 30 21:50:51.832947 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.832891 5801 cpu_manager.go:414] "containerMap: removing container" podUID="a5559491-62e9-4637-9253-ac2edfccbd46" containerName="payload-pre-processing" Oct 30 21:50:51.832947 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.832915 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5559491-62e9-4637-9253-ac2edfccbd46" containerName="payload-pre-processing" Oct 30 21:50:51.833146 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.833061 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5559491-62e9-4637-9253-ac2edfccbd46" containerName="payload-pre-processing" Oct 30 21:50:51.839021 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.838975 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt" Oct 30 21:50:51.893100 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.893054 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt"] Oct 30 21:50:51.920004 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.919911 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aff10ad-d945-4d00-b340-3dce95cdb26c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt\" (UID: \"9aff10ad-d945-4d00-b340-3dce95cdb26c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt" Oct 30 21:50:51.920004 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.919977 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aff10ad-d945-4d00-b340-3dce95cdb26c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt\" (UID: \"9aff10ad-d945-4d00-b340-3dce95cdb26c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt" Oct 30 21:50:51.920214 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.920111 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnnff\" (UniqueName: \"kubernetes.io/projected/9aff10ad-d945-4d00-b340-3dce95cdb26c-kube-api-access-tnnff\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt\" (UID: \"9aff10ad-d945-4d00-b340-3dce95cdb26c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt" Oct 30 21:50:51.935031 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.935003 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7" Oct 30 21:50:51.949274 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:51.949252 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg" Oct 30 21:50:52.018824 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.018789 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw"] Oct 30 21:50:52.024277 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.024243 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9d5666d-c9d2-4570-a859-82622af82893-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d5666d-c9d2-4570-a859-82622af82893-maas-api-xks-ca\") pod \"e9d5666d-c9d2-4570-a859-82622af82893\" (UID: \"e9d5666d-c9d2-4570-a859-82622af82893\") " Oct 30 21:50:52.024490 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.024474 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9d5666d-c9d2-4570-a859-82622af82893-kube-api-access-ml29c\" (UniqueName: \"kubernetes.io/projected/e9d5666d-c9d2-4570-a859-82622af82893-kube-api-access-ml29c\") pod \"e9d5666d-c9d2-4570-a859-82622af82893\" (UID: \"e9d5666d-c9d2-4570-a859-82622af82893\") " Oct 30 21:50:52.024742 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.024728 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9d5666d-c9d2-4570-a859-82622af82893-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d5666d-c9d2-4570-a859-82622af82893-maas-api-openshift-ca\") pod \"e9d5666d-c9d2-4570-a859-82622af82893\" (UID: \"e9d5666d-c9d2-4570-a859-82622af82893\") " Oct 30 21:50:52.025234 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.025210 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aff10ad-d945-4d00-b340-3dce95cdb26c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt\" (UID: \"9aff10ad-d945-4d00-b340-3dce95cdb26c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt" Oct 30 21:50:52.025405 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.025387 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aff10ad-d945-4d00-b340-3dce95cdb26c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt\" (UID: \"9aff10ad-d945-4d00-b340-3dce95cdb26c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt" Oct 30 21:50:52.025554 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.025539 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnnff\" (UniqueName: \"kubernetes.io/projected/9aff10ad-d945-4d00-b340-3dce95cdb26c-kube-api-access-tnnff\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt\" (UID: \"9aff10ad-d945-4d00-b340-3dce95cdb26c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt" Oct 30 21:50:52.026142 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.025297 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbf428-665f98f4bc-8wzhw"] Oct 30 21:50:52.026417 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.026391 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aff10ad-d945-4d00-b340-3dce95cdb26c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt\" (UID: \"9aff10ad-d945-4d00-b340-3dce95cdb26c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt" Oct 30 21:50:52.026789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.026746 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d5666d-c9d2-4570-a859-82622af82893-maas-api-openshift-ca" pod "e9d5666d-c9d2-4570-a859-82622af82893" (UID: "e9d5666d-c9d2-4570-a859-82622af82893"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:52.027840 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.027815 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d5666d-c9d2-4570-a859-82622af82893-maas-api-xks-ca" pod "e9d5666d-c9d2-4570-a859-82622af82893" (UID: "e9d5666d-c9d2-4570-a859-82622af82893"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:52.028391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.028375 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aff10ad-d945-4d00-b340-3dce95cdb26c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt\" (UID: \"9aff10ad-d945-4d00-b340-3dce95cdb26c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt" Oct 30 21:50:52.028934 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.028916 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d5666d-c9d2-4570-a859-82622af82893-kube-api-access-ml29c" pod "e9d5666d-c9d2-4570-a859-82622af82893" (UID: "e9d5666d-c9d2-4570-a859-82622af82893"). InnerVolumeSpecName "kube-api-access-ml29c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:52.082398 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.082285 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnnff\" (UniqueName: \"kubernetes.io/projected/9aff10ad-d945-4d00-b340-3dce95cdb26c-kube-api-access-tnnff\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt\" (UID: \"9aff10ad-d945-4d00-b340-3dce95cdb26c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt" Oct 30 21:50:52.127285 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.127253 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d5666d-c9d2-4570-a859-82622af82893-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:52.127285 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.127283 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d5666d-c9d2-4570-a859-82622af82893-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:52.127459 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.127293 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml29c\" (UniqueName: \"kubernetes.io/projected/e9d5666d-c9d2-4570-a859-82622af82893-kube-api-access-ml29c\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:52.247590 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.247556 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt" Oct 30 21:50:52.269110 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.269078 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7"] Oct 30 21:50:52.270860 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:50:52.270838 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983f1693_9287_4523_a83f_70da14c30336.slice/crio-75f896b660127f4c60ca733f4aa1067bfac7878e2740cf9e346df2eed962bb28 WatchSource:0}: Error finding container 75f896b660127f4c60ca733f4aa1067bfac7878e2740cf9e346df2eed962bb28: Status 404 returned error can't find the container with id 75f896b660127f4c60ca733f4aa1067bfac7878e2740cf9e346df2eed962bb28 Oct 30 21:50:52.544397 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:50:52.544364 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aff10ad_d945_4d00_b340_3dce95cdb26c.slice/crio-c29669403a36a350df07ce067c434258c02aa63b65a21a17402838ed4b7a29a3 WatchSource:0}: Error finding container c29669403a36a350df07ce067c434258c02aa63b65a21a17402838ed4b7a29a3: Status 404 returned error can't find the container with id c29669403a36a350df07ce067c434258c02aa63b65a21a17402838ed4b7a29a3 Oct 30 21:50:52.544519 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.544438 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt"] Oct 30 21:50:52.675951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.675931 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg" Oct 30 21:50:52.676076 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.676010 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg" event={"ID":"e9d5666d-c9d2-4570-a859-82622af82893","Type":"ContainerDied","Data":"4e900a0bcd2eba1d8d78490953003d43d4e0fec8aa317d0e110d945b494e9709"} Oct 30 21:50:52.676076 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.676062 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e900a0bcd2eba1d8d78490953003d43d4e0fec8aa317d0e110d945b494e9709" Oct 30 21:50:52.677927 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.677899 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7" event={"ID":"983f1693-9287-4523-a83f-70da14c30336","Type":"ContainerStarted","Data":"75f896b660127f4c60ca733f4aa1067bfac7878e2740cf9e346df2eed962bb28"} Oct 30 21:50:52.680996 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.680960 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt" event={"ID":"9aff10ad-d945-4d00-b340-3dce95cdb26c","Type":"ContainerStarted","Data":"c29669403a36a350df07ce067c434258c02aa63b65a21a17402838ed4b7a29a3"} Oct 30 21:50:52.699302 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.699258 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7" podStartSLOduration=1.699245746 podStartE2EDuration="1.699245746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:52.696491618 +0000 UTC m=+1321.292308280" watchObservedRunningTime="2026-10-30 21:50:52.699245746 +0000 UTC m=+1321.295062568" Oct 30 21:50:52.781004 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:52.780969 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:53.076586 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.076524 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:50:53.144877 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.144843 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-workload-socket\") pod \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " Oct 30 21:50:53.145039 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.144926 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-token\" (UniqueName: \"kubernetes.io/projected/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-token\") pod \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " Oct 30 21:50:53.145039 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.144949 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-podinfo\") pod \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " Oct 30 21:50:53.145039 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.144986 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-envoy\") pod \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " Oct 30 21:50:53.145221 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.145065 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-credential-socket\") pod \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " Oct 30 21:50:53.145221 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.145100 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-workload-certs\") pod \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " Oct 30 21:50:53.145221 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.145157 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-workload-socket" pod "0e91a97c-e981-4ab6-b6cd-3f92381ed5c4" (UID: "0e91a97c-e981-4ab6-b6cd-3f92381ed5c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:53.145221 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.145188 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istiod-ca-cert\") pod \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " Oct 30 21:50:53.145221 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.145212 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-data\") pod \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " Oct 30 21:50:53.145491 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.145263 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-kube-api-access-fv727\" (UniqueName: \"kubernetes.io/projected/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-kube-api-access-fv727\") pod \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\" (UID: \"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4\") " Oct 30 21:50:53.145600 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.145577 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-workload-certs" pod "0e91a97c-e981-4ab6-b6cd-3f92381ed5c4" (UID: "0e91a97c-e981-4ab6-b6cd-3f92381ed5c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:53.145812 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.145785 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-credential-socket" pod "0e91a97c-e981-4ab6-b6cd-3f92381ed5c4" (UID: "0e91a97c-e981-4ab6-b6cd-3f92381ed5c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:53.146047 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.146029 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:53.146430 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.146172 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:53.146430 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.146083 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istiod-ca-cert" pod "0e91a97c-e981-4ab6-b6cd-3f92381ed5c4" (UID: "0e91a97c-e981-4ab6-b6cd-3f92381ed5c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:53.146430 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.146302 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-data" pod "0e91a97c-e981-4ab6-b6cd-3f92381ed5c4" (UID: "0e91a97c-e981-4ab6-b6cd-3f92381ed5c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:53.147997 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.147691 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-envoy" pod "0e91a97c-e981-4ab6-b6cd-3f92381ed5c4" (UID: "0e91a97c-e981-4ab6-b6cd-3f92381ed5c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:53.148102 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.147990 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-kube-api-access-fv727" pod "0e91a97c-e981-4ab6-b6cd-3f92381ed5c4" (UID: "0e91a97c-e981-4ab6-b6cd-3f92381ed5c4"). InnerVolumeSpecName "kube-api-access-fv727". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:53.148102 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.148058 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-token" pod "0e91a97c-e981-4ab6-b6cd-3f92381ed5c4" (UID: "0e91a97c-e981-4ab6-b6cd-3f92381ed5c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:53.148345 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.148318 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-podinfo" pod "0e91a97c-e981-4ab6-b6cd-3f92381ed5c4" (UID: "0e91a97c-e981-4ab6-b6cd-3f92381ed5c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:53.247237 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.247211 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:53.247237 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.247237 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:53.247397 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.247247 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv727\" (UniqueName: \"kubernetes.io/projected/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-kube-api-access-fv727\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:53.247397 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.247256 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:53.247397 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.247265 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:53.247397 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.247273 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:53.247397 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.247281 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:53.527887 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.527803 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5559491-62e9-4637-9253-ac2edfccbd46" path="/var/lib/kubelet/pods/a5559491-62e9-4637-9253-ac2edfccbd46/volumes" Oct 30 21:50:53.528472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.528449 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb686228-0c5b-4d64-82b4-1838e0f75369" path="/var/lib/kubelet/pods/cb686228-0c5b-4d64-82b4-1838e0f75369/volumes" Oct 30 21:50:53.535344 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.535274 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:53.693774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.693721 5801 generic.go:356] "Generic (PLEG): container finished" podID="0e91a97c-e981-4ab6-b6cd-3f92381ed5c4" containerID="6200424ff91b7331861a8103fd76e92a79a228b40526b3f653f80e377e873dcf" exitCode=0 Oct 30 21:50:53.693971 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.693825 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" Oct 30 21:50:53.693971 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.693792 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" event={"ID":"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4","Type":"ContainerDied","Data":"6200424ff91b7331861a8103fd76e92a79a228b40526b3f653f80e377e873dcf"} Oct 30 21:50:53.693971 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.693948 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf" event={"ID":"0e91a97c-e981-4ab6-b6cd-3f92381ed5c4","Type":"ContainerDied","Data":"c93df807ae8964af71a3409aa6a1648256ff8e7c83955465ddb73e28e4ad28a0"} Oct 30 21:50:53.694135 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.693975 5801 scope.go:122] "RemoveContainer" containerID="6200424ff91b7331861a8103fd76e92a79a228b40526b3f653f80e377e873dcf" Oct 30 21:50:53.695912 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.695885 5801 generic.go:356] "Generic (PLEG): container finished" podID="983f1693-9287-4523-a83f-70da14c30336" containerID="645609fad69b6467636bf55484e520127c99d109ea7d5b07fc0ef96e2d46ae29" exitCode=0 Oct 30 21:50:53.696042 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.695967 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7" event={"ID":"983f1693-9287-4523-a83f-70da14c30336","Type":"ContainerDied","Data":"645609fad69b6467636bf55484e520127c99d109ea7d5b07fc0ef96e2d46ae29"} Oct 30 21:50:53.698349 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.698325 5801 generic.go:356] "Generic (PLEG): container finished" podID="9aff10ad-d945-4d00-b340-3dce95cdb26c" containerID="d646ec5ac332821664a6f420f4778cacfcf3f94d36d05b9154cef80d81ea128b" exitCode=0 Oct 30 21:50:53.698450 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.698376 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt" event={"ID":"9aff10ad-d945-4d00-b340-3dce95cdb26c","Type":"ContainerDied","Data":"d646ec5ac332821664a6f420f4778cacfcf3f94d36d05b9154cef80d81ea128b"} Oct 30 21:50:53.706745 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.706728 5801 scope.go:122] "RemoveContainer" containerID="6200424ff91b7331861a8103fd76e92a79a228b40526b3f653f80e377e873dcf" Oct 30 21:50:53.707057 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:50:53.707037 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6200424ff91b7331861a8103fd76e92a79a228b40526b3f653f80e377e873dcf\": container with ID starting with 6200424ff91b7331861a8103fd76e92a79a228b40526b3f653f80e377e873dcf not found: ID does not exist" containerID="6200424ff91b7331861a8103fd76e92a79a228b40526b3f653f80e377e873dcf" Oct 30 21:50:53.707133 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.707064 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6200424ff91b7331861a8103fd76e92a79a228b40526b3f653f80e377e873dcf"} err="failed to get container status \"6200424ff91b7331861a8103fd76e92a79a228b40526b3f653f80e377e873dcf\": rpc error: code = NotFound desc = could not find container \"6200424ff91b7331861a8103fd76e92a79a228b40526b3f653f80e377e873dcf\": container with ID starting with 6200424ff91b7331861a8103fd76e92a79a228b40526b3f653f80e377e873dcf not found: ID does not exist" Oct 30 21:50:53.762396 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.762359 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf"] Oct 30 21:50:53.767363 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:53.767335 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f8144d-openshift-default-5b96d887-msbqf"] Oct 30 21:50:54.295646 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:54.295560 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:54.710949 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:54.710903 5801 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:50:54.711351 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:54.710968 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" podUID="2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:50:54.871709 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:54.871677 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7" Oct 30 21:50:54.933816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:54.932742 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt" Oct 30 21:50:54.979606 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:54.979510 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/983f1693-9287-4523-a83f-70da14c30336-kube-api-access-spggj\" (UniqueName: \"kubernetes.io/projected/983f1693-9287-4523-a83f-70da14c30336-kube-api-access-spggj\") pod \"983f1693-9287-4523-a83f-70da14c30336\" (UID: \"983f1693-9287-4523-a83f-70da14c30336\") " Oct 30 21:50:54.979606 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:54.979594 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aff10ad-d945-4d00-b340-3dce95cdb26c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aff10ad-d945-4d00-b340-3dce95cdb26c-maas-api-xks-ca\") pod \"9aff10ad-d945-4d00-b340-3dce95cdb26c\" (UID: \"9aff10ad-d945-4d00-b340-3dce95cdb26c\") " Oct 30 21:50:54.979882 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:54.979687 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/983f1693-9287-4523-a83f-70da14c30336-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983f1693-9287-4523-a83f-70da14c30336-maas-api-xks-ca\") pod \"983f1693-9287-4523-a83f-70da14c30336\" (UID: \"983f1693-9287-4523-a83f-70da14c30336\") " Oct 30 21:50:54.980786 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:54.980745 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aff10ad-d945-4d00-b340-3dce95cdb26c-maas-api-openshift-ca" pod "9aff10ad-d945-4d00-b340-3dce95cdb26c" (UID: "9aff10ad-d945-4d00-b340-3dce95cdb26c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:54.980954 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:54.980932 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aff10ad-d945-4d00-b340-3dce95cdb26c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aff10ad-d945-4d00-b340-3dce95cdb26c-maas-api-openshift-ca\") pod \"9aff10ad-d945-4d00-b340-3dce95cdb26c\" (UID: \"9aff10ad-d945-4d00-b340-3dce95cdb26c\") " Oct 30 21:50:54.981703 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:54.981624 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aff10ad-d945-4d00-b340-3dce95cdb26c-kube-api-access-tnnff\" (UniqueName: \"kubernetes.io/projected/9aff10ad-d945-4d00-b340-3dce95cdb26c-kube-api-access-tnnff\") pod \"9aff10ad-d945-4d00-b340-3dce95cdb26c\" (UID: \"9aff10ad-d945-4d00-b340-3dce95cdb26c\") " Oct 30 21:50:54.982198 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:54.982152 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/983f1693-9287-4523-a83f-70da14c30336-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983f1693-9287-4523-a83f-70da14c30336-maas-api-openshift-ca\") pod \"983f1693-9287-4523-a83f-70da14c30336\" (UID: \"983f1693-9287-4523-a83f-70da14c30336\") " Oct 30 21:50:54.982298 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:54.982062 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aff10ad-d945-4d00-b340-3dce95cdb26c-maas-api-xks-ca" pod "9aff10ad-d945-4d00-b340-3dce95cdb26c" (UID: "9aff10ad-d945-4d00-b340-3dce95cdb26c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:54.982473 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:54.982446 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/983f1693-9287-4523-a83f-70da14c30336-maas-api-openshift-ca" pod "983f1693-9287-4523-a83f-70da14c30336" (UID: "983f1693-9287-4523-a83f-70da14c30336"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:54.982985 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:54.982959 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aff10ad-d945-4d00-b340-3dce95cdb26c-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:54.983076 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:54.982985 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983f1693-9287-4523-a83f-70da14c30336-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:54.983076 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:54.983000 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aff10ad-d945-4d00-b340-3dce95cdb26c-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:54.984087 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:54.984066 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aff10ad-d945-4d00-b340-3dce95cdb26c-kube-api-access-tnnff" pod "9aff10ad-d945-4d00-b340-3dce95cdb26c" (UID: "9aff10ad-d945-4d00-b340-3dce95cdb26c"). InnerVolumeSpecName "kube-api-access-tnnff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:54.984280 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:54.984261 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/983f1693-9287-4523-a83f-70da14c30336-maas-api-xks-ca" pod "983f1693-9287-4523-a83f-70da14c30336" (UID: "983f1693-9287-4523-a83f-70da14c30336"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:54.985450 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:54.985430 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983f1693-9287-4523-a83f-70da14c30336-kube-api-access-spggj" pod "983f1693-9287-4523-a83f-70da14c30336" (UID: "983f1693-9287-4523-a83f-70da14c30336"). InnerVolumeSpecName "kube-api-access-spggj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:54.986232 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:54.986206 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:55.011700 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.011662 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr"] Oct 30 21:50:55.012550 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.012525 5801 cpu_manager.go:414] "containerMap: removing container" podUID="9aff10ad-d945-4d00-b340-3dce95cdb26c" containerName="revoke-keys" Oct 30 21:50:55.012641 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.012551 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aff10ad-d945-4d00-b340-3dce95cdb26c" containerName="revoke-keys" Oct 30 21:50:55.012641 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.012578 5801 cpu_manager.go:414] "containerMap: removing container" podUID="e9d5666d-c9d2-4570-a859-82622af82893" containerName="revoke-keys" Oct 30 21:50:55.012641 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.012585 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d5666d-c9d2-4570-a859-82622af82893" containerName="revoke-keys" Oct 30 21:50:55.012641 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.012635 5801 cpu_manager.go:414] "containerMap: removing container" podUID="983f1693-9287-4523-a83f-70da14c30336" containerName="revoke-keys" Oct 30 21:50:55.012964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.012644 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="983f1693-9287-4523-a83f-70da14c30336" containerName="revoke-keys" Oct 30 21:50:55.012964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.012653 5801 cpu_manager.go:414] "containerMap: removing container" podUID="0e91a97c-e981-4ab6-b6cd-3f92381ed5c4" containerName="istio-proxy" Oct 30 21:50:55.012964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.012659 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e91a97c-e981-4ab6-b6cd-3f92381ed5c4" containerName="istio-proxy" Oct 30 21:50:55.012964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.012829 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aff10ad-d945-4d00-b340-3dce95cdb26c" containerName="revoke-keys" Oct 30 21:50:55.012964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.012844 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d5666d-c9d2-4570-a859-82622af82893" containerName="revoke-keys" Oct 30 21:50:55.012964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.012861 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="983f1693-9287-4523-a83f-70da14c30336" containerName="revoke-keys" Oct 30 21:50:55.012964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.012872 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e91a97c-e981-4ab6-b6cd-3f92381ed5c4" containerName="istio-proxy" Oct 30 21:50:55.017208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.017185 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr" Oct 30 21:50:55.028236 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.028210 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr"] Oct 30 21:50:55.083690 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.083654 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2rmc\" (UniqueName: \"kubernetes.io/projected/71bea698-5d69-4cce-8803-3080fe7ca730-kube-api-access-m2rmc\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr\" (UID: \"71bea698-5d69-4cce-8803-3080fe7ca730\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr" Oct 30 21:50:55.083868 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.083725 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71bea698-5d69-4cce-8803-3080fe7ca730-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr\" (UID: \"71bea698-5d69-4cce-8803-3080fe7ca730\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr" Oct 30 21:50:55.083918 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.083889 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71bea698-5d69-4cce-8803-3080fe7ca730-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr\" (UID: \"71bea698-5d69-4cce-8803-3080fe7ca730\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr" Oct 30 21:50:55.084090 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.084072 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spggj\" (UniqueName: \"kubernetes.io/projected/983f1693-9287-4523-a83f-70da14c30336-kube-api-access-spggj\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:55.084129 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.084095 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983f1693-9287-4523-a83f-70da14c30336-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:55.084129 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.084106 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnnff\" (UniqueName: \"kubernetes.io/projected/9aff10ad-d945-4d00-b340-3dce95cdb26c-kube-api-access-tnnff\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:55.192677 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.192633 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71bea698-5d69-4cce-8803-3080fe7ca730-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr\" (UID: \"71bea698-5d69-4cce-8803-3080fe7ca730\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr" Oct 30 21:50:55.210850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.198426 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2rmc\" (UniqueName: \"kubernetes.io/projected/71bea698-5d69-4cce-8803-3080fe7ca730-kube-api-access-m2rmc\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr\" (UID: \"71bea698-5d69-4cce-8803-3080fe7ca730\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr" Oct 30 21:50:55.210850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.198561 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71bea698-5d69-4cce-8803-3080fe7ca730-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr\" (UID: \"71bea698-5d69-4cce-8803-3080fe7ca730\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr" Oct 30 21:50:55.210850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.199401 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71bea698-5d69-4cce-8803-3080fe7ca730-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr\" (UID: \"71bea698-5d69-4cce-8803-3080fe7ca730\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr" Oct 30 21:50:55.210850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.207920 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71bea698-5d69-4cce-8803-3080fe7ca730-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr\" (UID: \"71bea698-5d69-4cce-8803-3080fe7ca730\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr" Oct 30 21:50:55.214003 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.213979 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2rmc\" (UniqueName: \"kubernetes.io/projected/71bea698-5d69-4cce-8803-3080fe7ca730-kube-api-access-m2rmc\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr\" (UID: \"71bea698-5d69-4cce-8803-3080fe7ca730\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr" Oct 30 21:50:55.215138 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.215114 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:55.318093 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.318055 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:55.345020 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.344987 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr" Oct 30 21:50:55.374060 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.373850 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx"] Oct 30 21:50:55.392095 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.392048 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fbf428-vc7hx"] Oct 30 21:50:55.525690 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.525652 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e91a97c-e981-4ab6-b6cd-3f92381ed5c4" path="/var/lib/kubelet/pods/0e91a97c-e981-4ab6-b6cd-3f92381ed5c4/volumes" Oct 30 21:50:55.526169 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.526148 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d74e114c-82f3-4540-9f23-8bf7cbd317f3" path="/var/lib/kubelet/pods/d74e114c-82f3-4540-9f23-8bf7cbd317f3/volumes" Oct 30 21:50:55.551563 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.551408 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:55.655329 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.655303 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr"] Oct 30 21:50:55.656814 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:50:55.656740 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71bea698_5d69_4cce_8803_3080fe7ca730.slice/crio-e2ee693139632bdf9fe297465c4df630ae407bf3ed4140ea7f1be0bdbf6b67bf WatchSource:0}: Error finding container e2ee693139632bdf9fe297465c4df630ae407bf3ed4140ea7f1be0bdbf6b67bf: Status 404 returned error can't find the container with id e2ee693139632bdf9fe297465c4df630ae407bf3ed4140ea7f1be0bdbf6b67bf Oct 30 21:50:55.719464 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.719425 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr" event={"ID":"71bea698-5d69-4cce-8803-3080fe7ca730","Type":"ContainerStarted","Data":"e2ee693139632bdf9fe297465c4df630ae407bf3ed4140ea7f1be0bdbf6b67bf"} Oct 30 21:50:55.721043 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.721019 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7" Oct 30 21:50:55.721043 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.721030 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7" event={"ID":"983f1693-9287-4523-a83f-70da14c30336","Type":"ContainerDied","Data":"75f896b660127f4c60ca733f4aa1067bfac7878e2740cf9e346df2eed962bb28"} Oct 30 21:50:55.721243 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.721069 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75f896b660127f4c60ca733f4aa1067bfac7878e2740cf9e346df2eed962bb28" Oct 30 21:50:55.722516 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.722497 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt" Oct 30 21:50:55.722516 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.722507 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt" event={"ID":"9aff10ad-d945-4d00-b340-3dce95cdb26c","Type":"ContainerDied","Data":"c29669403a36a350df07ce067c434258c02aa63b65a21a17402838ed4b7a29a3"} Oct 30 21:50:55.722673 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.722555 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c29669403a36a350df07ce067c434258c02aa63b65a21a17402838ed4b7a29a3" Oct 30 21:50:55.906592 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:55.906489 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:56.115554 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.115528 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:50:56.221182 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.221151 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-podinfo\") pod \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " Oct 30 21:50:56.224225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.221211 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-kube-api-access-nfvtd\" (UniqueName: \"kubernetes.io/projected/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-kube-api-access-nfvtd\") pod \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " Oct 30 21:50:56.224225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.221271 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-workload-socket\") pod \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " Oct 30 21:50:56.224225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.221303 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-envoy\") pod \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " Oct 30 21:50:56.224225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.221429 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-data\") pod \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " Oct 30 21:50:56.224225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.221564 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-workload-socket" pod "a7e5545a-7bc6-437b-bef6-4c6ee6776f09" (UID: "a7e5545a-7bc6-437b-bef6-4c6ee6776f09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:56.224225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.221579 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-workload-certs\") pod \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " Oct 30 21:50:56.224225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.221630 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-token\" (UniqueName: \"kubernetes.io/projected/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-token\") pod \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " Oct 30 21:50:56.224225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.221657 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istiod-ca-cert\") pod \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " Oct 30 21:50:56.224225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.221685 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-data" pod "a7e5545a-7bc6-437b-bef6-4c6ee6776f09" (UID: "a7e5545a-7bc6-437b-bef6-4c6ee6776f09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:56.224225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.221696 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-credential-socket\") pod \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\" (UID: \"a7e5545a-7bc6-437b-bef6-4c6ee6776f09\") " Oct 30 21:50:56.224225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.221906 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-workload-certs" pod "a7e5545a-7bc6-437b-bef6-4c6ee6776f09" (UID: "a7e5545a-7bc6-437b-bef6-4c6ee6776f09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:56.224225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.222005 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-credential-socket" pod "a7e5545a-7bc6-437b-bef6-4c6ee6776f09" (UID: "a7e5545a-7bc6-437b-bef6-4c6ee6776f09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:56.224225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.222325 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istiod-ca-cert" pod "a7e5545a-7bc6-437b-bef6-4c6ee6776f09" (UID: "a7e5545a-7bc6-437b-bef6-4c6ee6776f09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:56.224225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.222534 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.224225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.222555 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.224225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.222572 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.224225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.222587 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.224225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.222600 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.224225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.224194 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-token" pod "a7e5545a-7bc6-437b-bef6-4c6ee6776f09" (UID: "a7e5545a-7bc6-437b-bef6-4c6ee6776f09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:56.225051 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.224270 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-kube-api-access-nfvtd" pod "a7e5545a-7bc6-437b-bef6-4c6ee6776f09" (UID: "a7e5545a-7bc6-437b-bef6-4c6ee6776f09"). InnerVolumeSpecName "kube-api-access-nfvtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:56.225051 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.224463 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-envoy" pod "a7e5545a-7bc6-437b-bef6-4c6ee6776f09" (UID: "a7e5545a-7bc6-437b-bef6-4c6ee6776f09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:56.225051 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.224642 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-podinfo" pod "a7e5545a-7bc6-437b-bef6-4c6ee6776f09" (UID: "a7e5545a-7bc6-437b-bef6-4c6ee6776f09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:56.323926 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.323900 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.323926 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.323928 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.324081 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.323938 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.324081 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.323947 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfvtd\" (UniqueName: \"kubernetes.io/projected/a7e5545a-7bc6-437b-bef6-4c6ee6776f09-kube-api-access-nfvtd\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:56.434355 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.434281 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:56.732187 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.732155 5801 generic.go:356] "Generic (PLEG): container finished" podID="2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" containerID="afdce3adb37cb696660601a7973845154500045189525ba635e7bc274a0c5636" exitCode=0 Oct 30 21:50:56.732609 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.732228 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" event={"ID":"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe","Type":"ContainerDied","Data":"afdce3adb37cb696660601a7973845154500045189525ba635e7bc274a0c5636"} Oct 30 21:50:56.732727 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.732709 5801 scope.go:122] "RemoveContainer" containerID="afdce3adb37cb696660601a7973845154500045189525ba635e7bc274a0c5636" Oct 30 21:50:56.734151 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.734124 5801 generic.go:356] "Generic (PLEG): container finished" podID="71bea698-5d69-4cce-8803-3080fe7ca730" containerID="4c12a67a508d4cdb6c0fb0e5f38d50c3cbafd97adc877430570bee98f43b1a14" exitCode=0 Oct 30 21:50:56.734271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.734199 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr" event={"ID":"71bea698-5d69-4cce-8803-3080fe7ca730","Type":"ContainerDied","Data":"4c12a67a508d4cdb6c0fb0e5f38d50c3cbafd97adc877430570bee98f43b1a14"} Oct 30 21:50:56.734338 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.734271 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:50:56.734338 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.734333 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:50:56.734427 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.734366 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:50:56.736806 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.736784 5801 generic.go:356] "Generic (PLEG): container finished" podID="a7e5545a-7bc6-437b-bef6-4c6ee6776f09" containerID="1e072289193d11c9d353f41dabe560444a8737a005c53be22a79c809d6764f16" exitCode=0 Oct 30 21:50:56.736911 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.736848 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" Oct 30 21:50:56.736990 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.736903 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" event={"ID":"a7e5545a-7bc6-437b-bef6-4c6ee6776f09","Type":"ContainerDied","Data":"1e072289193d11c9d353f41dabe560444a8737a005c53be22a79c809d6764f16"} Oct 30 21:50:56.736990 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.736953 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5" event={"ID":"a7e5545a-7bc6-437b-bef6-4c6ee6776f09","Type":"ContainerDied","Data":"bab9ccb8e3e498268d8e8fb577678f17ce3d0a9973a6ce19ea8510ce2b808941"} Oct 30 21:50:56.736990 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.736983 5801 scope.go:122] "RemoveContainer" containerID="1e072289193d11c9d353f41dabe560444a8737a005c53be22a79c809d6764f16" Oct 30 21:50:56.754440 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.754418 5801 scope.go:122] "RemoveContainer" containerID="d944e9bb4c96c7dcf933f124619b6d599bee4f73f83a6c6712a109b6f60cfb72" Oct 30 21:50:56.766814 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.766791 5801 scope.go:122] "RemoveContainer" containerID="1e072289193d11c9d353f41dabe560444a8737a005c53be22a79c809d6764f16" Oct 30 21:50:56.767118 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:50:56.767098 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e072289193d11c9d353f41dabe560444a8737a005c53be22a79c809d6764f16\": container with ID starting with 1e072289193d11c9d353f41dabe560444a8737a005c53be22a79c809d6764f16 not found: ID does not exist" containerID="1e072289193d11c9d353f41dabe560444a8737a005c53be22a79c809d6764f16" Oct 30 21:50:56.767200 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.767123 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e072289193d11c9d353f41dabe560444a8737a005c53be22a79c809d6764f16"} err="failed to get container status \"1e072289193d11c9d353f41dabe560444a8737a005c53be22a79c809d6764f16\": rpc error: code = NotFound desc = could not find container \"1e072289193d11c9d353f41dabe560444a8737a005c53be22a79c809d6764f16\": container with ID starting with 1e072289193d11c9d353f41dabe560444a8737a005c53be22a79c809d6764f16 not found: ID does not exist" Oct 30 21:50:56.767200 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.767140 5801 scope.go:122] "RemoveContainer" containerID="d944e9bb4c96c7dcf933f124619b6d599bee4f73f83a6c6712a109b6f60cfb72" Oct 30 21:50:56.767429 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:50:56.767399 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d944e9bb4c96c7dcf933f124619b6d599bee4f73f83a6c6712a109b6f60cfb72\": container with ID starting with d944e9bb4c96c7dcf933f124619b6d599bee4f73f83a6c6712a109b6f60cfb72 not found: ID does not exist" containerID="d944e9bb4c96c7dcf933f124619b6d599bee4f73f83a6c6712a109b6f60cfb72" Oct 30 21:50:56.767483 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.767438 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d944e9bb4c96c7dcf933f124619b6d599bee4f73f83a6c6712a109b6f60cfb72"} err="failed to get container status \"d944e9bb4c96c7dcf933f124619b6d599bee4f73f83a6c6712a109b6f60cfb72\": rpc error: code = NotFound desc = could not find container \"d944e9bb4c96c7dcf933f124619b6d599bee4f73f83a6c6712a109b6f60cfb72\": container with ID starting with d944e9bb4c96c7dcf933f124619b6d599bee4f73f83a6c6712a109b6f60cfb72 not found: ID does not exist" Oct 30 21:50:56.787795 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.781006 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5"] Oct 30 21:50:56.787795 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:56.786505 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6cf5f0-openshift-default-75ff764fb7-6sln5"] Oct 30 21:50:57.164225 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:57.164194 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:57.525505 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:57.525470 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7e5545a-7bc6-437b-bef6-4c6ee6776f09" path="/var/lib/kubelet/pods/a7e5545a-7bc6-437b-bef6-4c6ee6776f09/volumes" Oct 30 21:50:57.770318 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:57.770272 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" event={"ID":"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe","Type":"ContainerStarted","Data":"9344a4e7bc82e3469071ac0829511178629abe6f71e15860c4900d40dca58d23"} Oct 30 21:50:57.917854 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:57.917820 5801 scope.go:122] "RemoveContainer" containerID="c85347b278842a802f33182da10d8fa1810acca660fde1dae0f0020c40002795" Oct 30 21:50:57.924522 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:57.924477 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:57.930578 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:57.930561 5801 scope.go:122] "RemoveContainer" containerID="3729e0fa8b1c0b35e967e0f299fbd02d628c80651e073448b3007d5b2108d8a0" Oct 30 21:50:57.994743 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:57.993530 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr" Oct 30 21:50:57.996958 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:57.996930 5801 scope.go:122] "RemoveContainer" containerID="c136f0caf0208ce9494c36ce7c024a1280db31ec7ff6130258c7bde4b55e827a" Oct 30 21:50:58.047148 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:58.047116 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71bea698-5d69-4cce-8803-3080fe7ca730-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71bea698-5d69-4cce-8803-3080fe7ca730-maas-api-xks-ca\") pod \"71bea698-5d69-4cce-8803-3080fe7ca730\" (UID: \"71bea698-5d69-4cce-8803-3080fe7ca730\") " Oct 30 21:50:58.047292 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:58.047221 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71bea698-5d69-4cce-8803-3080fe7ca730-kube-api-access-m2rmc\" (UniqueName: \"kubernetes.io/projected/71bea698-5d69-4cce-8803-3080fe7ca730-kube-api-access-m2rmc\") pod \"71bea698-5d69-4cce-8803-3080fe7ca730\" (UID: \"71bea698-5d69-4cce-8803-3080fe7ca730\") " Oct 30 21:50:58.047365 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:58.047319 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71bea698-5d69-4cce-8803-3080fe7ca730-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71bea698-5d69-4cce-8803-3080fe7ca730-maas-api-openshift-ca\") pod \"71bea698-5d69-4cce-8803-3080fe7ca730\" (UID: \"71bea698-5d69-4cce-8803-3080fe7ca730\") " Oct 30 21:50:58.048055 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:58.048012 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71bea698-5d69-4cce-8803-3080fe7ca730-maas-api-openshift-ca" pod "71bea698-5d69-4cce-8803-3080fe7ca730" (UID: "71bea698-5d69-4cce-8803-3080fe7ca730"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:58.049708 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:58.049668 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71bea698-5d69-4cce-8803-3080fe7ca730-maas-api-xks-ca" pod "71bea698-5d69-4cce-8803-3080fe7ca730" (UID: "71bea698-5d69-4cce-8803-3080fe7ca730"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:58.050480 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:58.050455 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71bea698-5d69-4cce-8803-3080fe7ca730-kube-api-access-m2rmc" pod "71bea698-5d69-4cce-8803-3080fe7ca730" (UID: "71bea698-5d69-4cce-8803-3080fe7ca730"). InnerVolumeSpecName "kube-api-access-m2rmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:58.136048 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:58.135396 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:58.149100 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:58.149071 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71bea698-5d69-4cce-8803-3080fe7ca730-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:58.149100 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:58.149099 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71bea698-5d69-4cce-8803-3080fe7ca730-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:58.149283 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:58.149120 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2rmc\" (UniqueName: \"kubernetes.io/projected/71bea698-5d69-4cce-8803-3080fe7ca730-kube-api-access-m2rmc\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:58.483235 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:58.483195 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:50:58.483490 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:58.483413 5801 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 21:50:58.483490 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:58.483456 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" podUID="2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 21:50:58.647735 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:58.647693 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:58.700974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:58.700928 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9"] Oct 30 21:50:58.710796 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:58.710709 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84"] Oct 30 21:50:58.798660 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:58.798570 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:58.808467 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:58.808440 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr" Oct 30 21:50:58.808696 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:58.808535 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr" event={"ID":"71bea698-5d69-4cce-8803-3080fe7ca730","Type":"ContainerDied","Data":"e2ee693139632bdf9fe297465c4df630ae407bf3ed4140ea7f1be0bdbf6b67bf"} Oct 30 21:50:58.808696 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:58.808565 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2ee693139632bdf9fe297465c4df630ae407bf3ed4140ea7f1be0bdbf6b67bf" Oct 30 21:50:58.897959 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:58.897924 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:59.483579 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:59.483548 5801 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 21:50:59.483785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:59.483600 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" podUID="2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 21:50:59.512267 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:50:59.512233 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:00.302847 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:00.302809 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:51:00.372323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:00.372228 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:00.491247 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:00.491208 5801 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 21:51:00.491456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:00.491264 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" podUID="2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 21:51:01.151045 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:01.150996 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:01.483964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:01.483869 5801 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 21:51:01.483964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:01.483937 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" podUID="2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 21:51:01.527275 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:01.527244 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh"] Oct 30 21:51:01.600804 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:01.600258 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:01.697028 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:01.696990 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:01.851520 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:01.851487 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:02.110801 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:02.110696 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:02.210717 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:02.210683 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:02.497083 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:02.496989 5801 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 21:51:02.497948 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:02.497914 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" podUID="2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 21:51:02.732008 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:02.731966 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:03.352257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:03.352218 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:03.451558 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:03.451521 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:03.607036 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:03.606952 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:51:03.607480 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:03.607384 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:51:03.608316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:03.608292 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:51:03.701154 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:03.701123 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq"] Oct 30 21:51:03.702006 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:03.701967 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9" podUID="99a88411-dade-4b2f-98a1-24675c120cad" containerName="payload-processing" containerID="cri-o://da21626418113e945967fcf70fbf4accbeba0123b684af4440575a4508f26d78" gracePeriod=25 Oct 30 21:51:03.711642 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:03.711615 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84" podUID="e95bb984-7abc-49e6-acb7-5250d1b4d98c" containerName="payload-pre-processing" containerID="cri-o://4446252aa2c572e360073011dc8d4218313f52c9f0b26083c86ab92023c86d71" gracePeriod=25 Oct 30 21:51:03.740580 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:03.740555 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:03.876441 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:03.874332 5801 generic.go:356] "Generic (PLEG): container finished" podID="e95bb984-7abc-49e6-acb7-5250d1b4d98c" containerID="4446252aa2c572e360073011dc8d4218313f52c9f0b26083c86ab92023c86d71" exitCode=0 Oct 30 21:51:03.876441 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:03.874487 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84" event={"ID":"e95bb984-7abc-49e6-acb7-5250d1b4d98c","Type":"ContainerDied","Data":"4446252aa2c572e360073011dc8d4218313f52c9f0b26083c86ab92023c86d71"} Oct 30 21:51:03.889479 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:03.889434 5801 generic.go:356] "Generic (PLEG): container finished" podID="99a88411-dade-4b2f-98a1-24675c120cad" containerID="da21626418113e945967fcf70fbf4accbeba0123b684af4440575a4508f26d78" exitCode=0 Oct 30 21:51:03.889895 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:03.889814 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9" event={"ID":"99a88411-dade-4b2f-98a1-24675c120cad","Type":"ContainerDied","Data":"da21626418113e945967fcf70fbf4accbeba0123b684af4440575a4508f26d78"} Oct 30 21:51:03.894489 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:03.890544 5801 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" secret="" err="secret \"e2e-shared-b-w2-fbf428-openshift-default-dockercfg-cqlbx\" not found" Oct 30 21:51:04.081250 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.081229 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9" Oct 30 21:51:04.097367 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.097348 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84" Oct 30 21:51:04.133113 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.133044 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99a88411-dade-4b2f-98a1-24675c120cad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a88411-dade-4b2f-98a1-24675c120cad-plugins-config-volume\") pod \"99a88411-dade-4b2f-98a1-24675c120cad\" (UID: \"99a88411-dade-4b2f-98a1-24675c120cad\") " Oct 30 21:51:04.133268 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.133249 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a88411-dade-4b2f-98a1-24675c120cad-kube-api-access-jdz7l\" (UniqueName: \"kubernetes.io/projected/99a88411-dade-4b2f-98a1-24675c120cad-kube-api-access-jdz7l\") pod \"99a88411-dade-4b2f-98a1-24675c120cad\" (UID: \"99a88411-dade-4b2f-98a1-24675c120cad\") " Oct 30 21:51:04.133488 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.133463 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a88411-dade-4b2f-98a1-24675c120cad-plugins-config-volume" pod "99a88411-dade-4b2f-98a1-24675c120cad" (UID: "99a88411-dade-4b2f-98a1-24675c120cad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:04.133734 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.133716 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a88411-dade-4b2f-98a1-24675c120cad-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:04.135283 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.135262 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a88411-dade-4b2f-98a1-24675c120cad-kube-api-access-jdz7l" pod "99a88411-dade-4b2f-98a1-24675c120cad" (UID: "99a88411-dade-4b2f-98a1-24675c120cad"). InnerVolumeSpecName "kube-api-access-jdz7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:04.225828 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.225795 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:04.234234 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.234212 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e95bb984-7abc-49e6-acb7-5250d1b4d98c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95bb984-7abc-49e6-acb7-5250d1b4d98c-plugins-config-volume\") pod \"e95bb984-7abc-49e6-acb7-5250d1b4d98c\" (UID: \"e95bb984-7abc-49e6-acb7-5250d1b4d98c\") " Oct 30 21:51:04.234436 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.234420 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e95bb984-7abc-49e6-acb7-5250d1b4d98c-kube-api-access-pgcvp\" (UniqueName: \"kubernetes.io/projected/e95bb984-7abc-49e6-acb7-5250d1b4d98c-kube-api-access-pgcvp\") pod \"e95bb984-7abc-49e6-acb7-5250d1b4d98c\" (UID: \"e95bb984-7abc-49e6-acb7-5250d1b4d98c\") " Oct 30 21:51:04.234556 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.234536 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95bb984-7abc-49e6-acb7-5250d1b4d98c-plugins-config-volume" pod "e95bb984-7abc-49e6-acb7-5250d1b4d98c" (UID: "e95bb984-7abc-49e6-acb7-5250d1b4d98c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:04.234917 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.234898 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdz7l\" (UniqueName: \"kubernetes.io/projected/99a88411-dade-4b2f-98a1-24675c120cad-kube-api-access-jdz7l\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:04.235000 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.234921 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95bb984-7abc-49e6-acb7-5250d1b4d98c-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:04.236373 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.236351 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e95bb984-7abc-49e6-acb7-5250d1b4d98c-kube-api-access-pgcvp" pod "e95bb984-7abc-49e6-acb7-5250d1b4d98c" (UID: "e95bb984-7abc-49e6-acb7-5250d1b4d98c"). InnerVolumeSpecName "kube-api-access-pgcvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:04.325928 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.325893 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:04.336388 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.336358 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgcvp\" (UniqueName: \"kubernetes.io/projected/e95bb984-7abc-49e6-acb7-5250d1b4d98c-kube-api-access-pgcvp\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:04.400008 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.399928 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5"] Oct 30 21:51:04.400477 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.400463 5801 cpu_manager.go:414] "containerMap: removing container" podUID="a7e5545a-7bc6-437b-bef6-4c6ee6776f09" containerName="istio-proxy" Oct 30 21:51:04.400525 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.400478 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e5545a-7bc6-437b-bef6-4c6ee6776f09" containerName="istio-proxy" Oct 30 21:51:04.400525 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.400492 5801 cpu_manager.go:414] "containerMap: removing container" podUID="e95bb984-7abc-49e6-acb7-5250d1b4d98c" containerName="payload-pre-processing" Oct 30 21:51:04.400525 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.400497 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e95bb984-7abc-49e6-acb7-5250d1b4d98c" containerName="payload-pre-processing" Oct 30 21:51:04.400525 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.400512 5801 cpu_manager.go:414] "containerMap: removing container" podUID="71bea698-5d69-4cce-8803-3080fe7ca730" containerName="revoke-keys" Oct 30 21:51:04.400525 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.400517 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71bea698-5d69-4cce-8803-3080fe7ca730" containerName="revoke-keys" Oct 30 21:51:04.400678 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.400529 5801 cpu_manager.go:414] "containerMap: removing container" podUID="99a88411-dade-4b2f-98a1-24675c120cad" containerName="payload-processing" Oct 30 21:51:04.400678 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.400533 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a88411-dade-4b2f-98a1-24675c120cad" containerName="payload-processing" Oct 30 21:51:04.400678 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.400550 5801 cpu_manager.go:414] "containerMap: removing container" podUID="a7e5545a-7bc6-437b-bef6-4c6ee6776f09" containerName="istio-proxy" Oct 30 21:51:04.400678 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.400554 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e5545a-7bc6-437b-bef6-4c6ee6776f09" containerName="istio-proxy" Oct 30 21:51:04.400678 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.400634 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e5545a-7bc6-437b-bef6-4c6ee6776f09" containerName="istio-proxy" Oct 30 21:51:04.400678 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.400644 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="e95bb984-7abc-49e6-acb7-5250d1b4d98c" containerName="payload-pre-processing" Oct 30 21:51:04.400678 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.400651 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="71bea698-5d69-4cce-8803-3080fe7ca730" containerName="revoke-keys" Oct 30 21:51:04.400678 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.400658 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a88411-dade-4b2f-98a1-24675c120cad" containerName="payload-processing" Oct 30 21:51:04.404302 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.404286 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5" Oct 30 21:51:04.408649 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.408623 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:51:04.408791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.408742 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:51:04.408875 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.408836 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:51:04.408986 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.408967 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:51:04.414103 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.414082 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5"] Oct 30 21:51:04.538881 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.538850 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87025c3e-8014-4283-8f26-32822e095d93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5\" (UID: \"87025c3e-8014-4283-8f26-32822e095d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5" Oct 30 21:51:04.539054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.538980 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87025c3e-8014-4283-8f26-32822e095d93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5\" (UID: \"87025c3e-8014-4283-8f26-32822e095d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5" Oct 30 21:51:04.539054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.539033 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc7p4\" (UniqueName: \"kubernetes.io/projected/87025c3e-8014-4283-8f26-32822e095d93-kube-api-access-wc7p4\") pod \"maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5\" (UID: \"87025c3e-8014-4283-8f26-32822e095d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5" Oct 30 21:51:04.640086 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.640051 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87025c3e-8014-4283-8f26-32822e095d93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5\" (UID: \"87025c3e-8014-4283-8f26-32822e095d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5" Oct 30 21:51:04.640543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.640141 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87025c3e-8014-4283-8f26-32822e095d93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5\" (UID: \"87025c3e-8014-4283-8f26-32822e095d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5" Oct 30 21:51:04.640543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.640171 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc7p4\" (UniqueName: \"kubernetes.io/projected/87025c3e-8014-4283-8f26-32822e095d93-kube-api-access-wc7p4\") pod \"maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5\" (UID: \"87025c3e-8014-4283-8f26-32822e095d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5" Oct 30 21:51:04.640722 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.640700 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87025c3e-8014-4283-8f26-32822e095d93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5\" (UID: \"87025c3e-8014-4283-8f26-32822e095d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5" Oct 30 21:51:04.642601 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.642575 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87025c3e-8014-4283-8f26-32822e095d93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5\" (UID: \"87025c3e-8014-4283-8f26-32822e095d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5" Oct 30 21:51:04.652121 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.652067 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc7p4\" (UniqueName: \"kubernetes.io/projected/87025c3e-8014-4283-8f26-32822e095d93-kube-api-access-wc7p4\") pod \"maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5\" (UID: \"87025c3e-8014-4283-8f26-32822e095d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5" Oct 30 21:51:04.721383 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.721356 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5" Oct 30 21:51:04.904303 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.904207 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84" event={"ID":"e95bb984-7abc-49e6-acb7-5250d1b4d98c","Type":"ContainerDied","Data":"6281ac2c83001faba83daadc590d8d49bc63d93ba377e1f33f282075ed23f8c1"} Oct 30 21:51:04.904303 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.904286 5801 scope.go:122] "RemoveContainer" containerID="4446252aa2c572e360073011dc8d4218313f52c9f0b26083c86ab92023c86d71" Oct 30 21:51:04.904532 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.904433 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84" Oct 30 21:51:04.906694 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.906662 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" podUID="2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" containerName="istio-proxy" containerID="cri-o://9344a4e7bc82e3469071ac0829511178629abe6f71e15860c4900d40dca58d23" gracePeriod=30 Oct 30 21:51:04.907197 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.907001 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9" event={"ID":"99a88411-dade-4b2f-98a1-24675c120cad","Type":"ContainerDied","Data":"164c55e5b89417c17b9486f518880eda9b7adf9ecc0d2f70d65f6173fc3e5555"} Oct 30 21:51:04.907197 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.907093 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9" Oct 30 21:51:04.922228 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.922206 5801 scope.go:122] "RemoveContainer" containerID="da21626418113e945967fcf70fbf4accbeba0123b684af4440575a4508f26d78" Oct 30 21:51:04.946635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.946605 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:04.959105 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.959076 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84"] Oct 30 21:51:04.963071 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.963048 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f608f7-d5f9bb96c-t2v84"] Oct 30 21:51:04.973731 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.973711 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9"] Oct 30 21:51:04.979069 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:04.979048 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f608f7-6f4568bf4c-qsvt9"] Oct 30 21:51:05.033378 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:05.033351 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5"] Oct 30 21:51:05.034458 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:51:05.034432 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87025c3e_8014_4283_8f26_32822e095d93.slice/crio-c18bb206a5cb9cb78039b07278737f5fe04010ea40f6bf6e0f7f63c9e2997973 WatchSource:0}: Error finding container c18bb206a5cb9cb78039b07278737f5fe04010ea40f6bf6e0f7f63c9e2997973: Status 404 returned error can't find the container with id c18bb206a5cb9cb78039b07278737f5fe04010ea40f6bf6e0f7f63c9e2997973 Oct 30 21:51:05.147776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:05.146488 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:05.256915 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:05.256568 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:05.299482 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:05.299450 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt"] Oct 30 21:51:05.303573 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:05.303546 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-dc3edf-wchqt"] Oct 30 21:51:05.360448 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:05.360406 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:05.461836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:05.461803 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:05.570336 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:05.565978 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99a88411-dade-4b2f-98a1-24675c120cad" path="/var/lib/kubelet/pods/99a88411-dade-4b2f-98a1-24675c120cad/volumes" Oct 30 21:51:05.570336 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:05.566445 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aff10ad-d945-4d00-b340-3dce95cdb26c" path="/var/lib/kubelet/pods/9aff10ad-d945-4d00-b340-3dce95cdb26c/volumes" Oct 30 21:51:05.570336 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:05.566893 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e95bb984-7abc-49e6-acb7-5250d1b4d98c" path="/var/lib/kubelet/pods/e95bb984-7abc-49e6-acb7-5250d1b4d98c/volumes" Oct 30 21:51:05.833667 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:05.833573 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:05.917154 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:05.917103 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5" event={"ID":"87025c3e-8014-4283-8f26-32822e095d93","Type":"ContainerStarted","Data":"913f2c100dcbc127d995574ca82eac06c781b8207d54cf828487d27566759afe"} Oct 30 21:51:05.917364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:05.917186 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5" event={"ID":"87025c3e-8014-4283-8f26-32822e095d93","Type":"ContainerStarted","Data":"c18bb206a5cb9cb78039b07278737f5fe04010ea40f6bf6e0f7f63c9e2997973"} Oct 30 21:51:05.940338 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:05.940286 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5" podStartSLOduration=1.940271987 podStartE2EDuration="1.940271987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:05.938119558 +0000 UTC m=+1334.533936219" watchObservedRunningTime="2026-10-30 21:51:05.940271987 +0000 UTC m=+1334.536088647" Oct 30 21:51:06.116941 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.116853 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:06.249874 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.249838 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:06.438888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.438775 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:06.442659 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.442617 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7"] Oct 30 21:51:06.458844 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.458816 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aad4dd-7pcx7"] Oct 30 21:51:06.528431 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.528381 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh" podUID="54e288ae-db61-46b6-884e-0c199fbacaa7" containerName="payload-processing" containerID="cri-o://cfe52991760ad0ea2cf18555ef13ec3495808680fbff4ce650a67d0642780fac" gracePeriod=25 Oct 30 21:51:06.589986 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.589947 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:06.824127 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.824103 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh" Oct 30 21:51:06.871390 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.871280 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:06.886984 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.886953 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54e288ae-db61-46b6-884e-0c199fbacaa7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54e288ae-db61-46b6-884e-0c199fbacaa7-plugins-config-volume\") pod \"54e288ae-db61-46b6-884e-0c199fbacaa7\" (UID: \"54e288ae-db61-46b6-884e-0c199fbacaa7\") " Oct 30 21:51:06.887172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.887153 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54e288ae-db61-46b6-884e-0c199fbacaa7-kube-api-access-9bsjx\" (UniqueName: \"kubernetes.io/projected/54e288ae-db61-46b6-884e-0c199fbacaa7-kube-api-access-9bsjx\") pod \"54e288ae-db61-46b6-884e-0c199fbacaa7\" (UID: \"54e288ae-db61-46b6-884e-0c199fbacaa7\") " Oct 30 21:51:06.887386 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.887361 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54e288ae-db61-46b6-884e-0c199fbacaa7-plugins-config-volume" pod "54e288ae-db61-46b6-884e-0c199fbacaa7" (UID: "54e288ae-db61-46b6-884e-0c199fbacaa7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:06.887726 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.887704 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54e288ae-db61-46b6-884e-0c199fbacaa7-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:06.889167 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.889145 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e288ae-db61-46b6-884e-0c199fbacaa7-kube-api-access-9bsjx" pod "54e288ae-db61-46b6-884e-0c199fbacaa7" (UID: "54e288ae-db61-46b6-884e-0c199fbacaa7"). InnerVolumeSpecName "kube-api-access-9bsjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:06.927491 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.927467 5801 generic.go:356] "Generic (PLEG): container finished" podID="87025c3e-8014-4283-8f26-32822e095d93" containerID="913f2c100dcbc127d995574ca82eac06c781b8207d54cf828487d27566759afe" exitCode=0 Oct 30 21:51:06.927590 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.927512 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5" event={"ID":"87025c3e-8014-4283-8f26-32822e095d93","Type":"ContainerDied","Data":"913f2c100dcbc127d995574ca82eac06c781b8207d54cf828487d27566759afe"} Oct 30 21:51:06.928972 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.928949 5801 generic.go:356] "Generic (PLEG): container finished" podID="54e288ae-db61-46b6-884e-0c199fbacaa7" containerID="cfe52991760ad0ea2cf18555ef13ec3495808680fbff4ce650a67d0642780fac" exitCode=0 Oct 30 21:51:06.929055 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.928978 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh" event={"ID":"54e288ae-db61-46b6-884e-0c199fbacaa7","Type":"ContainerDied","Data":"cfe52991760ad0ea2cf18555ef13ec3495808680fbff4ce650a67d0642780fac"} Oct 30 21:51:06.929055 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.929002 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh" Oct 30 21:51:06.929055 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.929017 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh" event={"ID":"54e288ae-db61-46b6-884e-0c199fbacaa7","Type":"ContainerDied","Data":"37f0a21586b7039bc7ad0672c845bfd78ed85557c8a348832a766639bf44cc70"} Oct 30 21:51:06.929055 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.929039 5801 scope.go:122] "RemoveContainer" containerID="cfe52991760ad0ea2cf18555ef13ec3495808680fbff4ce650a67d0642780fac" Oct 30 21:51:06.939916 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.939833 5801 scope.go:122] "RemoveContainer" containerID="cfe52991760ad0ea2cf18555ef13ec3495808680fbff4ce650a67d0642780fac" Oct 30 21:51:06.940083 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:06.940064 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfe52991760ad0ea2cf18555ef13ec3495808680fbff4ce650a67d0642780fac\": container with ID starting with cfe52991760ad0ea2cf18555ef13ec3495808680fbff4ce650a67d0642780fac not found: ID does not exist" containerID="cfe52991760ad0ea2cf18555ef13ec3495808680fbff4ce650a67d0642780fac" Oct 30 21:51:06.940143 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.940097 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfe52991760ad0ea2cf18555ef13ec3495808680fbff4ce650a67d0642780fac"} err="failed to get container status \"cfe52991760ad0ea2cf18555ef13ec3495808680fbff4ce650a67d0642780fac\": rpc error: code = NotFound desc = could not find container \"cfe52991760ad0ea2cf18555ef13ec3495808680fbff4ce650a67d0642780fac\": container with ID starting with cfe52991760ad0ea2cf18555ef13ec3495808680fbff4ce650a67d0642780fac not found: ID does not exist" Oct 30 21:51:06.965190 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.965141 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh"] Oct 30 21:51:06.969447 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.969428 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-aad4dd-66dfc5f8c9-gkfrh"] Oct 30 21:51:06.988881 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:06.988860 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bsjx\" (UniqueName: \"kubernetes.io/projected/54e288ae-db61-46b6-884e-0c199fbacaa7-kube-api-access-9bsjx\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:07.484557 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:07.484519 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:07.529159 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:07.525907 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54e288ae-db61-46b6-884e-0c199fbacaa7" path="/var/lib/kubelet/pods/54e288ae-db61-46b6-884e-0c199fbacaa7/volumes" Oct 30 21:51:07.529347 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:07.529299 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="983f1693-9287-4523-a83f-70da14c30336" path="/var/lib/kubelet/pods/983f1693-9287-4523-a83f-70da14c30336/volumes" Oct 30 21:51:08.099721 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:08.099700 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5" Oct 30 21:51:08.190172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:08.190067 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:08.204324 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:08.204298 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87025c3e-8014-4283-8f26-32822e095d93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87025c3e-8014-4283-8f26-32822e095d93-maas-api-openshift-ca\") pod \"87025c3e-8014-4283-8f26-32822e095d93\" (UID: \"87025c3e-8014-4283-8f26-32822e095d93\") " Oct 30 21:51:08.204469 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:08.204350 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87025c3e-8014-4283-8f26-32822e095d93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87025c3e-8014-4283-8f26-32822e095d93-maas-api-xks-ca\") pod \"87025c3e-8014-4283-8f26-32822e095d93\" (UID: \"87025c3e-8014-4283-8f26-32822e095d93\") " Oct 30 21:51:08.204531 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:08.204470 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87025c3e-8014-4283-8f26-32822e095d93-kube-api-access-wc7p4\" (UniqueName: \"kubernetes.io/projected/87025c3e-8014-4283-8f26-32822e095d93-kube-api-access-wc7p4\") pod \"87025c3e-8014-4283-8f26-32822e095d93\" (UID: \"87025c3e-8014-4283-8f26-32822e095d93\") " Oct 30 21:51:08.204663 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:08.204641 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87025c3e-8014-4283-8f26-32822e095d93-maas-api-openshift-ca" pod "87025c3e-8014-4283-8f26-32822e095d93" (UID: "87025c3e-8014-4283-8f26-32822e095d93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:08.204956 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:08.204938 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87025c3e-8014-4283-8f26-32822e095d93-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:08.206229 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:08.206206 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87025c3e-8014-4283-8f26-32822e095d93-maas-api-xks-ca" pod "87025c3e-8014-4283-8f26-32822e095d93" (UID: "87025c3e-8014-4283-8f26-32822e095d93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:08.206361 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:08.206345 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87025c3e-8014-4283-8f26-32822e095d93-kube-api-access-wc7p4" pod "87025c3e-8014-4283-8f26-32822e095d93" (UID: "87025c3e-8014-4283-8f26-32822e095d93"). InnerVolumeSpecName "kube-api-access-wc7p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:08.306836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:08.306809 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87025c3e-8014-4283-8f26-32822e095d93-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:08.306836 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:08.306837 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc7p4\" (UniqueName: \"kubernetes.io/projected/87025c3e-8014-4283-8f26-32822e095d93-kube-api-access-wc7p4\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:08.354907 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:08.346174 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:08.446359 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:08.446281 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr"] Oct 30 21:51:08.447040 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:08.447014 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:08.449627 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:08.449606 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-0b412a-zkrrr"] Oct 30 21:51:08.547096 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:08.547065 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:08.956166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:08.956114 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5" event={"ID":"87025c3e-8014-4283-8f26-32822e095d93","Type":"ContainerDied","Data":"c18bb206a5cb9cb78039b07278737f5fe04010ea40f6bf6e0f7f63c9e2997973"} Oct 30 21:51:08.956389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:08.956231 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c18bb206a5cb9cb78039b07278737f5fe04010ea40f6bf6e0f7f63c9e2997973" Oct 30 21:51:08.956389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:08.956331 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5" Oct 30 21:51:09.081420 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:09.081385 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:09.190428 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:09.190396 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298"] Oct 30 21:51:09.351589 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:09.351557 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:09.458877 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:09.458841 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:09.529315 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:09.529284 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71bea698-5d69-4cce-8803-3080fe7ca730" path="/var/lib/kubelet/pods/71bea698-5d69-4cce-8803-3080fe7ca730/volumes" Oct 30 21:51:10.185668 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.185615 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:51:10.235310 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.235278 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-data\") pod \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " Oct 30 21:51:10.235763 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.235317 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-envoy\") pod \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " Oct 30 21:51:10.235763 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.235373 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-podinfo\") pod \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " Oct 30 21:51:10.235763 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.235484 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-credential-socket\") pod \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " Oct 30 21:51:10.235763 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.235536 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istiod-ca-cert\") pod \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " Oct 30 21:51:10.235763 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.235568 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-data" pod "2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" (UID: "2ea737f7-f6dc-4b40-8c4c-920a2131fcbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:10.235763 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.235619 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-workload-socket\") pod \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " Oct 30 21:51:10.235763 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.235654 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-kube-api-access-qgpht\" (UniqueName: \"kubernetes.io/projected/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-kube-api-access-qgpht\") pod \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " Oct 30 21:51:10.235763 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.235736 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-workload-certs\") pod \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " Oct 30 21:51:10.236271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.235794 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-token\" (UniqueName: \"kubernetes.io/projected/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-token\") pod \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\" (UID: \"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe\") " Oct 30 21:51:10.236271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.236007 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istiod-ca-cert" pod "2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" (UID: "2ea737f7-f6dc-4b40-8c4c-920a2131fcbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:10.236271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.236035 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-workload-certs" pod "2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" (UID: "2ea737f7-f6dc-4b40-8c4c-920a2131fcbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:10.237002 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.236150 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-credential-socket" pod "2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" (UID: "2ea737f7-f6dc-4b40-8c4c-920a2131fcbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:10.237153 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.236745 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-workload-socket" pod "2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" (UID: "2ea737f7-f6dc-4b40-8c4c-920a2131fcbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:10.237153 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.236500 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:10.237153 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.237053 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:10.237153 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.237072 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:10.238543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.238396 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-token" pod "2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" (UID: "2ea737f7-f6dc-4b40-8c4c-920a2131fcbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:10.238656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.238549 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-envoy" pod "2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" (UID: "2ea737f7-f6dc-4b40-8c4c-920a2131fcbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:10.238656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.238623 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-kube-api-access-qgpht" pod "2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" (UID: "2ea737f7-f6dc-4b40-8c4c-920a2131fcbe"). InnerVolumeSpecName "kube-api-access-qgpht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:10.238656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.238648 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-podinfo" pod "2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" (UID: "2ea737f7-f6dc-4b40-8c4c-920a2131fcbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:10.301079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.301036 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:10.338482 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.338439 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:10.338482 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.338478 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:10.338742 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.338494 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:10.338742 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.338510 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:10.338742 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.338531 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgpht\" (UniqueName: \"kubernetes.io/projected/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-kube-api-access-qgpht\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:10.338742 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.338547 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:10.975378 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.975342 5801 generic.go:356] "Generic (PLEG): container finished" podID="2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" containerID="9344a4e7bc82e3469071ac0829511178629abe6f71e15860c4900d40dca58d23" exitCode=0 Oct 30 21:51:10.975568 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.975415 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" Oct 30 21:51:10.975568 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.975426 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" event={"ID":"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe","Type":"ContainerDied","Data":"9344a4e7bc82e3469071ac0829511178629abe6f71e15860c4900d40dca58d23"} Oct 30 21:51:10.975568 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.975483 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq" event={"ID":"2ea737f7-f6dc-4b40-8c4c-920a2131fcbe","Type":"ContainerDied","Data":"89337231b67d5787e60862ab482ed3cbe94586940128510255e53fb15b8c86c5"} Oct 30 21:51:10.975568 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.975506 5801 scope.go:122] "RemoveContainer" containerID="9344a4e7bc82e3469071ac0829511178629abe6f71e15860c4900d40dca58d23" Oct 30 21:51:10.988221 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.987157 5801 scope.go:122] "RemoveContainer" containerID="afdce3adb37cb696660601a7973845154500045189525ba635e7bc274a0c5636" Oct 30 21:51:10.999179 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.999159 5801 scope.go:122] "RemoveContainer" containerID="9344a4e7bc82e3469071ac0829511178629abe6f71e15860c4900d40dca58d23" Oct 30 21:51:10.999441 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:10.999419 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9344a4e7bc82e3469071ac0829511178629abe6f71e15860c4900d40dca58d23\": container with ID starting with 9344a4e7bc82e3469071ac0829511178629abe6f71e15860c4900d40dca58d23 not found: ID does not exist" containerID="9344a4e7bc82e3469071ac0829511178629abe6f71e15860c4900d40dca58d23" Oct 30 21:51:10.999489 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.999450 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9344a4e7bc82e3469071ac0829511178629abe6f71e15860c4900d40dca58d23"} err="failed to get container status \"9344a4e7bc82e3469071ac0829511178629abe6f71e15860c4900d40dca58d23\": rpc error: code = NotFound desc = could not find container \"9344a4e7bc82e3469071ac0829511178629abe6f71e15860c4900d40dca58d23\": container with ID starting with 9344a4e7bc82e3469071ac0829511178629abe6f71e15860c4900d40dca58d23 not found: ID does not exist" Oct 30 21:51:10.999489 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.999466 5801 scope.go:122] "RemoveContainer" containerID="afdce3adb37cb696660601a7973845154500045189525ba635e7bc274a0c5636" Oct 30 21:51:10.999731 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:10.999712 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afdce3adb37cb696660601a7973845154500045189525ba635e7bc274a0c5636\": container with ID starting with afdce3adb37cb696660601a7973845154500045189525ba635e7bc274a0c5636 not found: ID does not exist" containerID="afdce3adb37cb696660601a7973845154500045189525ba635e7bc274a0c5636" Oct 30 21:51:10.999802 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:10.999736 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afdce3adb37cb696660601a7973845154500045189525ba635e7bc274a0c5636"} err="failed to get container status \"afdce3adb37cb696660601a7973845154500045189525ba635e7bc274a0c5636\": rpc error: code = NotFound desc = could not find container \"afdce3adb37cb696660601a7973845154500045189525ba635e7bc274a0c5636\": container with ID starting with afdce3adb37cb696660601a7973845154500045189525ba635e7bc274a0c5636 not found: ID does not exist" Oct 30 21:51:11.078975 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:11.078941 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq"] Oct 30 21:51:11.111306 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:11.111272 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fbf428-openshift-default-78876f7f7-nmvgq"] Oct 30 21:51:11.132158 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:11.132110 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:11.316777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:11.315540 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:11.355830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:11.355746 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg"] Oct 30 21:51:11.360124 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:11.360097 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f608f7-bx7hg"] Oct 30 21:51:11.422317 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:11.422279 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:11.526326 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:11.526292 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" path="/var/lib/kubelet/pods/2ea737f7-f6dc-4b40-8c4c-920a2131fcbe/volumes" Oct 30 21:51:11.526845 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:11.526801 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9d5666d-c9d2-4570-a859-82622af82893" path="/var/lib/kubelet/pods/e9d5666d-c9d2-4570-a859-82622af82893/volumes" Oct 30 21:51:11.743799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:11.743651 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:11.941960 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:11.941926 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:12.636439 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:12.636409 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:13.349159 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:13.349110 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:14.045917 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:14.045870 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:14.157795 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:14.157712 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:14.442550 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:14.442511 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft"] Oct 30 21:51:14.443496 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:14.443463 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" podUID="2fe17509-a317-4eae-8edd-de1383398615" containerName="istio-proxy" containerID="cri-o://c9fc4971acbfbd843a554bf516ba427b62331da91968c3ba02d505b5705d3f1b" gracePeriod=30 Oct 30 21:51:14.498012 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:14.497976 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:14.638224 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:14.638191 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:14.742219 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:14.742126 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:14.963257 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:14.963220 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:15.068113 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.068076 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:15.159586 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.159547 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84"] Oct 30 21:51:15.160130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.160112 5801 cpu_manager.go:414] "containerMap: removing container" podUID="54e288ae-db61-46b6-884e-0c199fbacaa7" containerName="payload-processing" Oct 30 21:51:15.160130 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.160129 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e288ae-db61-46b6-884e-0c199fbacaa7" containerName="payload-processing" Oct 30 21:51:15.160351 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.160143 5801 cpu_manager.go:414] "containerMap: removing container" podUID="2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" containerName="istio-proxy" Oct 30 21:51:15.160351 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.160149 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" containerName="istio-proxy" Oct 30 21:51:15.160351 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.160179 5801 cpu_manager.go:414] "containerMap: removing container" podUID="87025c3e-8014-4283-8f26-32822e095d93" containerName="revoke-keys" Oct 30 21:51:15.160351 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.160184 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87025c3e-8014-4283-8f26-32822e095d93" containerName="revoke-keys" Oct 30 21:51:15.160351 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.160269 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" containerName="istio-proxy" Oct 30 21:51:15.160351 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.160279 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e5545a-7bc6-437b-bef6-4c6ee6776f09" containerName="istio-proxy" Oct 30 21:51:15.160351 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.160288 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="87025c3e-8014-4283-8f26-32822e095d93" containerName="revoke-keys" Oct 30 21:51:15.160351 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.160296 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e288ae-db61-46b6-884e-0c199fbacaa7" containerName="payload-processing" Oct 30 21:51:15.165631 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.165606 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84" Oct 30 21:51:15.176065 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.176031 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84"] Oct 30 21:51:15.176244 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.176221 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:51:15.180855 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.180831 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:51:15.238882 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.229595 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84c6d7e-7ce7-4caf-97ec-43267070cab3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84\" (UID: \"c84c6d7e-7ce7-4caf-97ec-43267070cab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84" Oct 30 21:51:15.238882 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.229765 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84c6d7e-7ce7-4caf-97ec-43267070cab3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84\" (UID: \"c84c6d7e-7ce7-4caf-97ec-43267070cab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84" Oct 30 21:51:15.238882 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.229807 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt2gl\" (UniqueName: \"kubernetes.io/projected/c84c6d7e-7ce7-4caf-97ec-43267070cab3-kube-api-access-kt2gl\") pod \"maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84\" (UID: \"c84c6d7e-7ce7-4caf-97ec-43267070cab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84" Oct 30 21:51:15.238882 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.238161 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:51:15.238882 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.238295 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:51:15.330669 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.330581 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84c6d7e-7ce7-4caf-97ec-43267070cab3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84\" (UID: \"c84c6d7e-7ce7-4caf-97ec-43267070cab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84" Oct 30 21:51:15.330884 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.330732 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84c6d7e-7ce7-4caf-97ec-43267070cab3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84\" (UID: \"c84c6d7e-7ce7-4caf-97ec-43267070cab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84" Oct 30 21:51:15.330884 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.330796 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt2gl\" (UniqueName: \"kubernetes.io/projected/c84c6d7e-7ce7-4caf-97ec-43267070cab3-kube-api-access-kt2gl\") pod \"maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84\" (UID: \"c84c6d7e-7ce7-4caf-97ec-43267070cab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84" Oct 30 21:51:15.331233 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.331211 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:15.331676 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.331652 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84c6d7e-7ce7-4caf-97ec-43267070cab3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84\" (UID: \"c84c6d7e-7ce7-4caf-97ec-43267070cab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84" Oct 30 21:51:15.333168 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.333151 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84c6d7e-7ce7-4caf-97ec-43267070cab3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84\" (UID: \"c84c6d7e-7ce7-4caf-97ec-43267070cab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84" Oct 30 21:51:15.343866 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.343843 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt2gl\" (UniqueName: \"kubernetes.io/projected/c84c6d7e-7ce7-4caf-97ec-43267070cab3-kube-api-access-kt2gl\") pod \"maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84\" (UID: \"c84c6d7e-7ce7-4caf-97ec-43267070cab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84" Oct 30 21:51:15.432652 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.432621 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:15.554406 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.554068 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84" Oct 30 21:51:15.675612 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.675531 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz"] Oct 30 21:51:15.676055 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.676020 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" podUID="3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e" containerName="istio-proxy" containerID="cri-o://2234d70e1ea909adb643e94f3f7348c408d4b02151880f068245e2d826f44fc2" gracePeriod=30 Oct 30 21:51:15.712141 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.711014 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:15.917097 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:15.917069 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84"] Oct 30 21:51:15.917227 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:51:15.917151 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84c6d7e_7ce7_4caf_97ec_43267070cab3.slice/crio-75d53a70ca415bcb1ed0d925cd3eab60ecc401fdd191c13e8c8010c18d754b30 WatchSource:0}: Error finding container 75d53a70ca415bcb1ed0d925cd3eab60ecc401fdd191c13e8c8010c18d754b30: Status 404 returned error can't find the container with id 75d53a70ca415bcb1ed0d925cd3eab60ecc401fdd191c13e8c8010c18d754b30 Oct 30 21:51:16.017985 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:16.017952 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84" event={"ID":"c84c6d7e-7ce7-4caf-97ec-43267070cab3","Type":"ContainerStarted","Data":"75d53a70ca415bcb1ed0d925cd3eab60ecc401fdd191c13e8c8010c18d754b30"} Oct 30 21:51:16.274631 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:16.274577 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:16.376824 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:16.376785 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:16.378094 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:16.378072 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl"] Oct 30 21:51:16.378627 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:16.378614 5801 cpu_manager.go:414] "containerMap: removing container" podUID="2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" containerName="istio-proxy" Oct 30 21:51:16.378693 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:16.378629 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" containerName="istio-proxy" Oct 30 21:51:16.378729 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:16.378718 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea737f7-f6dc-4b40-8c4c-920a2131fcbe" containerName="istio-proxy" Oct 30 21:51:16.382201 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:16.382182 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl" Oct 30 21:51:16.392219 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:16.392196 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl"] Oct 30 21:51:16.565885 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:16.565858 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de2823f-1b5e-434e-987a-133d6f101aa6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl\" (UID: \"7de2823f-1b5e-434e-987a-133d6f101aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl" Oct 30 21:51:16.566053 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:16.565921 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de2823f-1b5e-434e-987a-133d6f101aa6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl\" (UID: \"7de2823f-1b5e-434e-987a-133d6f101aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl" Oct 30 21:51:16.566053 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:16.565994 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjxg2\" (UniqueName: \"kubernetes.io/projected/7de2823f-1b5e-434e-987a-133d6f101aa6-kube-api-access-wjxg2\") pod \"maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl\" (UID: \"7de2823f-1b5e-434e-987a-133d6f101aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl" Oct 30 21:51:16.585277 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:16.585246 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:16.668276 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:16.668154 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjxg2\" (UniqueName: \"kubernetes.io/projected/7de2823f-1b5e-434e-987a-133d6f101aa6-kube-api-access-wjxg2\") pod \"maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl\" (UID: \"7de2823f-1b5e-434e-987a-133d6f101aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl" Oct 30 21:51:16.668987 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:16.668955 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de2823f-1b5e-434e-987a-133d6f101aa6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl\" (UID: \"7de2823f-1b5e-434e-987a-133d6f101aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl" Oct 30 21:51:16.669128 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:16.669103 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de2823f-1b5e-434e-987a-133d6f101aa6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl\" (UID: \"7de2823f-1b5e-434e-987a-133d6f101aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl" Oct 30 21:51:16.670239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:16.670218 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de2823f-1b5e-434e-987a-133d6f101aa6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl\" (UID: \"7de2823f-1b5e-434e-987a-133d6f101aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl" Oct 30 21:51:16.672315 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:16.672287 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de2823f-1b5e-434e-987a-133d6f101aa6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl\" (UID: \"7de2823f-1b5e-434e-987a-133d6f101aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl" Oct 30 21:51:16.681545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:16.681514 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjxg2\" (UniqueName: \"kubernetes.io/projected/7de2823f-1b5e-434e-987a-133d6f101aa6-kube-api-access-wjxg2\") pod \"maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl\" (UID: \"7de2823f-1b5e-434e-987a-133d6f101aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl" Oct 30 21:51:16.698513 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:16.698492 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl" Oct 30 21:51:16.786082 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:16.786048 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:16.977302 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:16.977214 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:16.980169 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:16.980145 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl"] Oct 30 21:51:16.980909 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:51:16.980881 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de2823f_1b5e_434e_987a_133d6f101aa6.slice/crio-f236a7fc8361004617c9379c247df6fa94ec33d47aff77130d062f211980f298 WatchSource:0}: Error finding container f236a7fc8361004617c9379c247df6fa94ec33d47aff77130d062f211980f298: Status 404 returned error can't find the container with id f236a7fc8361004617c9379c247df6fa94ec33d47aff77130d062f211980f298 Oct 30 21:51:17.054774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:17.034713 5801 generic.go:356] "Generic (PLEG): container finished" podID="c84c6d7e-7ce7-4caf-97ec-43267070cab3" containerID="812eb8885d0d2f74739acda04038f7503cc4bf724a3f78123f52d3b7103087ae" exitCode=0 Oct 30 21:51:17.054774 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:17.035053 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84" event={"ID":"c84c6d7e-7ce7-4caf-97ec-43267070cab3","Type":"ContainerDied","Data":"812eb8885d0d2f74739acda04038f7503cc4bf724a3f78123f52d3b7103087ae"} Oct 30 21:51:17.069605 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:17.063141 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl" event={"ID":"7de2823f-1b5e-434e-987a-133d6f101aa6","Type":"ContainerStarted","Data":"f236a7fc8361004617c9379c247df6fa94ec33d47aff77130d062f211980f298"} Oct 30 21:51:17.269394 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:17.269290 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:17.450829 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:17.450793 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg"] Oct 30 21:51:17.455169 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:17.455143 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg" Oct 30 21:51:17.463344 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:17.463300 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg"] Oct 30 21:51:17.581183 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:17.581152 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed9e521-5edd-4d8c-886a-adc56eeb5637-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg\" (UID: \"9ed9e521-5edd-4d8c-886a-adc56eeb5637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg" Oct 30 21:51:17.581345 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:17.581215 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed9e521-5edd-4d8c-886a-adc56eeb5637-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg\" (UID: \"9ed9e521-5edd-4d8c-886a-adc56eeb5637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg" Oct 30 21:51:17.581345 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:17.581329 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbmk5\" (UniqueName: \"kubernetes.io/projected/9ed9e521-5edd-4d8c-886a-adc56eeb5637-kube-api-access-gbmk5\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg\" (UID: \"9ed9e521-5edd-4d8c-886a-adc56eeb5637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg" Oct 30 21:51:17.682810 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:17.682776 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed9e521-5edd-4d8c-886a-adc56eeb5637-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg\" (UID: \"9ed9e521-5edd-4d8c-886a-adc56eeb5637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg" Oct 30 21:51:17.683006 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:17.682917 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbmk5\" (UniqueName: \"kubernetes.io/projected/9ed9e521-5edd-4d8c-886a-adc56eeb5637-kube-api-access-gbmk5\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg\" (UID: \"9ed9e521-5edd-4d8c-886a-adc56eeb5637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg" Oct 30 21:51:17.683262 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:17.683193 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed9e521-5edd-4d8c-886a-adc56eeb5637-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg\" (UID: \"9ed9e521-5edd-4d8c-886a-adc56eeb5637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg" Oct 30 21:51:17.683726 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:17.683458 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed9e521-5edd-4d8c-886a-adc56eeb5637-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg\" (UID: \"9ed9e521-5edd-4d8c-886a-adc56eeb5637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg" Oct 30 21:51:17.685589 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:17.685560 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed9e521-5edd-4d8c-886a-adc56eeb5637-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg\" (UID: \"9ed9e521-5edd-4d8c-886a-adc56eeb5637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg" Oct 30 21:51:17.696995 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:17.696967 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbmk5\" (UniqueName: \"kubernetes.io/projected/9ed9e521-5edd-4d8c-886a-adc56eeb5637-kube-api-access-gbmk5\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg\" (UID: \"9ed9e521-5edd-4d8c-886a-adc56eeb5637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg" Oct 30 21:51:17.777798 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:17.777747 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg" Oct 30 21:51:17.961249 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:17.961121 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-760beae1-9k5tp"] Oct 30 21:51:17.965992 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:17.965965 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760beae1-9k5tp" Oct 30 21:51:17.973722 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:17.973689 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-760beae1-9k5tp"] Oct 30 21:51:18.060041 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.060009 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg"] Oct 30 21:51:18.060172 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:51:18.060129 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed9e521_5edd_4d8c_886a_adc56eeb5637.slice/crio-17857516fd943e1f66e6ddb6dfcb6f66dd1871f055870a05ab24d79885b041b1 WatchSource:0}: Error finding container 17857516fd943e1f66e6ddb6dfcb6f66dd1871f055870a05ab24d79885b041b1: Status 404 returned error can't find the container with id 17857516fd943e1f66e6ddb6dfcb6f66dd1871f055870a05ab24d79885b041b1 Oct 30 21:51:18.072643 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.072614 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg" event={"ID":"9ed9e521-5edd-4d8c-886a-adc56eeb5637","Type":"ContainerStarted","Data":"17857516fd943e1f66e6ddb6dfcb6f66dd1871f055870a05ab24d79885b041b1"} Oct 30 21:51:18.074476 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.074452 5801 generic.go:356] "Generic (PLEG): container finished" podID="7de2823f-1b5e-434e-987a-133d6f101aa6" containerID="caad361754218494e220b04a6e1a8dd7d684f86b593890f3abf426ef79890e59" exitCode=0 Oct 30 21:51:18.074604 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.074535 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl" event={"ID":"7de2823f-1b5e-434e-987a-133d6f101aa6","Type":"ContainerDied","Data":"caad361754218494e220b04a6e1a8dd7d684f86b593890f3abf426ef79890e59"} Oct 30 21:51:18.088334 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.088269 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-760beae1-9k5tp\" (UID: \"59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760beae1-9k5tp" Oct 30 21:51:18.088448 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.088354 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxgcx\" (UniqueName: \"kubernetes.io/projected/59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b-kube-api-access-sxgcx\") pod \"maas-api-revoke-sub-760beae1-9k5tp\" (UID: \"59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760beae1-9k5tp" Oct 30 21:51:18.088448 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.088441 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-760beae1-9k5tp\" (UID: \"59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760beae1-9k5tp" Oct 30 21:51:18.189856 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.189818 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-760beae1-9k5tp\" (UID: \"59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760beae1-9k5tp" Oct 30 21:51:18.190033 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.189977 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-760beae1-9k5tp\" (UID: \"59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760beae1-9k5tp" Oct 30 21:51:18.190119 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.190047 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxgcx\" (UniqueName: \"kubernetes.io/projected/59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b-kube-api-access-sxgcx\") pod \"maas-api-revoke-sub-760beae1-9k5tp\" (UID: \"59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760beae1-9k5tp" Oct 30 21:51:18.190770 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.190732 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-760beae1-9k5tp\" (UID: \"59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760beae1-9k5tp" Oct 30 21:51:18.192103 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.192080 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-760beae1-9k5tp\" (UID: \"59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760beae1-9k5tp" Oct 30 21:51:18.202288 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.202267 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxgcx\" (UniqueName: \"kubernetes.io/projected/59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b-kube-api-access-sxgcx\") pod \"maas-api-revoke-sub-760beae1-9k5tp\" (UID: \"59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760beae1-9k5tp" Oct 30 21:51:18.212550 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.212530 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84" Oct 30 21:51:18.240064 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.239984 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:18.323299 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.323265 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760beae1-9k5tp" Oct 30 21:51:18.396013 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.395619 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c84c6d7e-7ce7-4caf-97ec-43267070cab3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84c6d7e-7ce7-4caf-97ec-43267070cab3-maas-api-xks-ca\") pod \"c84c6d7e-7ce7-4caf-97ec-43267070cab3\" (UID: \"c84c6d7e-7ce7-4caf-97ec-43267070cab3\") " Oct 30 21:51:18.396013 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.395920 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c84c6d7e-7ce7-4caf-97ec-43267070cab3-kube-api-access-kt2gl\" (UniqueName: \"kubernetes.io/projected/c84c6d7e-7ce7-4caf-97ec-43267070cab3-kube-api-access-kt2gl\") pod \"c84c6d7e-7ce7-4caf-97ec-43267070cab3\" (UID: \"c84c6d7e-7ce7-4caf-97ec-43267070cab3\") " Oct 30 21:51:18.396201 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.396046 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c84c6d7e-7ce7-4caf-97ec-43267070cab3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84c6d7e-7ce7-4caf-97ec-43267070cab3-maas-api-openshift-ca\") pod \"c84c6d7e-7ce7-4caf-97ec-43267070cab3\" (UID: \"c84c6d7e-7ce7-4caf-97ec-43267070cab3\") " Oct 30 21:51:18.403383 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.396953 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84c6d7e-7ce7-4caf-97ec-43267070cab3-maas-api-openshift-ca" pod "c84c6d7e-7ce7-4caf-97ec-43267070cab3" (UID: "c84c6d7e-7ce7-4caf-97ec-43267070cab3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:18.403383 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.402558 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c84c6d7e-7ce7-4caf-97ec-43267070cab3-maas-api-xks-ca" pod "c84c6d7e-7ce7-4caf-97ec-43267070cab3" (UID: "c84c6d7e-7ce7-4caf-97ec-43267070cab3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:18.404162 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.404138 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84c6d7e-7ce7-4caf-97ec-43267070cab3-kube-api-access-kt2gl" pod "c84c6d7e-7ce7-4caf-97ec-43267070cab3" (UID: "c84c6d7e-7ce7-4caf-97ec-43267070cab3"). InnerVolumeSpecName "kube-api-access-kt2gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:18.498223 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.498192 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84c6d7e-7ce7-4caf-97ec-43267070cab3-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:18.498223 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.498221 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84c6d7e-7ce7-4caf-97ec-43267070cab3-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:18.498618 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.498235 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt2gl\" (UniqueName: \"kubernetes.io/projected/c84c6d7e-7ce7-4caf-97ec-43267070cab3-kube-api-access-kt2gl\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:18.615614 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.615524 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-760beae1-9k5tp"] Oct 30 21:51:18.615737 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:51:18.615708 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a8b3a0_eecb_48a1_9ee3_d94ac73ae63b.slice/crio-71e6aa4279b3962a6ac136ced5bc979c87f90e1d1be14ce0910586ca6c8dd91b WatchSource:0}: Error finding container 71e6aa4279b3962a6ac136ced5bc979c87f90e1d1be14ce0910586ca6c8dd91b: Status 404 returned error can't find the container with id 71e6aa4279b3962a6ac136ced5bc979c87f90e1d1be14ce0910586ca6c8dd91b Oct 30 21:51:18.837067 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.837036 5801 patch_prober.go:29] interesting pod/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:51:18.837210 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.837082 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" podUID="2fe17509-a317-4eae-8edd-de1383398615" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:51:18.937389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:18.937349 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:19.085269 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.085243 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84" Oct 30 21:51:19.085417 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.085268 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84" event={"ID":"c84c6d7e-7ce7-4caf-97ec-43267070cab3","Type":"ContainerDied","Data":"75d53a70ca415bcb1ed0d925cd3eab60ecc401fdd191c13e8c8010c18d754b30"} Oct 30 21:51:19.085417 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.085322 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75d53a70ca415bcb1ed0d925cd3eab60ecc401fdd191c13e8c8010c18d754b30" Oct 30 21:51:19.087541 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.087510 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760beae1-9k5tp" event={"ID":"59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b","Type":"ContainerStarted","Data":"39ecc5d7f9f6bb1230e29c4e604d282222c9a4c48b1fc03391cabc2cbb1c4bce"} Oct 30 21:51:19.087643 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.087571 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760beae1-9k5tp" event={"ID":"59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b","Type":"ContainerStarted","Data":"71e6aa4279b3962a6ac136ced5bc979c87f90e1d1be14ce0910586ca6c8dd91b"} Oct 30 21:51:19.089383 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.089360 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg" event={"ID":"9ed9e521-5edd-4d8c-886a-adc56eeb5637","Type":"ContainerStarted","Data":"1517550c5d3c80484620bbf73ecd70169c7210328d84c66729bdf39facb74269"} Oct 30 21:51:19.107308 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.107258 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760beae1-9k5tp" podStartSLOduration=2.107245822 podStartE2EDuration="2.107245822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:19.105166506 +0000 UTC m=+1347.700983166" watchObservedRunningTime="2026-10-30 21:51:19.107245822 +0000 UTC m=+1347.703062482" Oct 30 21:51:19.127417 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.127370 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg" podStartSLOduration=2.127356181 podStartE2EDuration="2.127356181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:19.123038486 +0000 UTC m=+1347.718855279" watchObservedRunningTime="2026-10-30 21:51:19.127356181 +0000 UTC m=+1347.723172851" Oct 30 21:51:19.213545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.213510 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl" Oct 30 21:51:19.409969 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.409895 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7de2823f-1b5e-434e-987a-133d6f101aa6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de2823f-1b5e-434e-987a-133d6f101aa6-maas-api-xks-ca\") pod \"7de2823f-1b5e-434e-987a-133d6f101aa6\" (UID: \"7de2823f-1b5e-434e-987a-133d6f101aa6\") " Oct 30 21:51:19.410123 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.410103 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7de2823f-1b5e-434e-987a-133d6f101aa6-kube-api-access-wjxg2\" (UniqueName: \"kubernetes.io/projected/7de2823f-1b5e-434e-987a-133d6f101aa6-kube-api-access-wjxg2\") pod \"7de2823f-1b5e-434e-987a-133d6f101aa6\" (UID: \"7de2823f-1b5e-434e-987a-133d6f101aa6\") " Oct 30 21:51:19.410265 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.410249 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7de2823f-1b5e-434e-987a-133d6f101aa6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de2823f-1b5e-434e-987a-133d6f101aa6-maas-api-openshift-ca\") pod \"7de2823f-1b5e-434e-987a-133d6f101aa6\" (UID: \"7de2823f-1b5e-434e-987a-133d6f101aa6\") " Oct 30 21:51:19.410614 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.410581 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7de2823f-1b5e-434e-987a-133d6f101aa6-maas-api-openshift-ca" pod "7de2823f-1b5e-434e-987a-133d6f101aa6" (UID: "7de2823f-1b5e-434e-987a-133d6f101aa6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:19.410867 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.410841 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de2823f-1b5e-434e-987a-133d6f101aa6-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:19.412107 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.412084 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7de2823f-1b5e-434e-987a-133d6f101aa6-maas-api-xks-ca" pod "7de2823f-1b5e-434e-987a-133d6f101aa6" (UID: "7de2823f-1b5e-434e-987a-133d6f101aa6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:19.412407 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.412382 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de2823f-1b5e-434e-987a-133d6f101aa6-kube-api-access-wjxg2" pod "7de2823f-1b5e-434e-987a-133d6f101aa6" (UID: "7de2823f-1b5e-434e-987a-133d6f101aa6"). InnerVolumeSpecName "kube-api-access-wjxg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:19.511960 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.511934 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de2823f-1b5e-434e-987a-133d6f101aa6-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:19.511960 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.511959 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjxg2\" (UniqueName: \"kubernetes.io/projected/7de2823f-1b5e-434e-987a-133d6f101aa6-kube-api-access-wjxg2\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:19.697896 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.697746 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:19.709202 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.709176 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:51:19.713277 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.713242 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-credential-socket\") pod \"2fe17509-a317-4eae-8edd-de1383398615\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " Oct 30 21:51:19.713398 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.713280 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-istio-data\") pod \"2fe17509-a317-4eae-8edd-de1383398615\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " Oct 30 21:51:19.713398 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.713322 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-workload-certs\") pod \"2fe17509-a317-4eae-8edd-de1383398615\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " Oct 30 21:51:19.713511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.713397 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-workload-socket\") pod \"2fe17509-a317-4eae-8edd-de1383398615\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " Oct 30 21:51:19.713722 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.713558 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-credential-socket" pod "2fe17509-a317-4eae-8edd-de1383398615" (UID: "2fe17509-a317-4eae-8edd-de1383398615"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:19.713811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.713706 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fe17509-a317-4eae-8edd-de1383398615-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fe17509-a317-4eae-8edd-de1383398615-istiod-ca-cert\") pod \"2fe17509-a317-4eae-8edd-de1383398615\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " Oct 30 21:51:19.713811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.713570 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-istio-data" pod "2fe17509-a317-4eae-8edd-de1383398615" (UID: "2fe17509-a317-4eae-8edd-de1383398615"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:19.713811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.713650 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-workload-socket" pod "2fe17509-a317-4eae-8edd-de1383398615" (UID: "2fe17509-a317-4eae-8edd-de1383398615"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:19.713811 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.713705 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-workload-certs" pod "2fe17509-a317-4eae-8edd-de1383398615" (UID: "2fe17509-a317-4eae-8edd-de1383398615"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:19.713976 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.713876 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fe17509-a317-4eae-8edd-de1383398615-kube-api-access-4t9n4\" (UniqueName: \"kubernetes.io/projected/2fe17509-a317-4eae-8edd-de1383398615-kube-api-access-4t9n4\") pod \"2fe17509-a317-4eae-8edd-de1383398615\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " Oct 30 21:51:19.713976 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.713939 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fe17509-a317-4eae-8edd-de1383398615-istiod-ca-cert" pod "2fe17509-a317-4eae-8edd-de1383398615" (UID: "2fe17509-a317-4eae-8edd-de1383398615"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:19.714090 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.713989 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fe17509-a317-4eae-8edd-de1383398615-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fe17509-a317-4eae-8edd-de1383398615-istio-podinfo\") pod \"2fe17509-a317-4eae-8edd-de1383398615\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " Oct 30 21:51:19.714090 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.714050 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-istio-envoy\") pod \"2fe17509-a317-4eae-8edd-de1383398615\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " Oct 30 21:51:19.714196 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.714103 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fe17509-a317-4eae-8edd-de1383398615-istio-token\" (UniqueName: \"kubernetes.io/projected/2fe17509-a317-4eae-8edd-de1383398615-istio-token\") pod \"2fe17509-a317-4eae-8edd-de1383398615\" (UID: \"2fe17509-a317-4eae-8edd-de1383398615\") " Oct 30 21:51:19.714706 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.714681 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:19.714835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.714709 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:19.714835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.714726 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:19.714835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.714742 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:19.714835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.714771 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fe17509-a317-4eae-8edd-de1383398615-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:19.716433 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.716410 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-istio-envoy" pod "2fe17509-a317-4eae-8edd-de1383398615" (UID: "2fe17509-a317-4eae-8edd-de1383398615"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:19.716517 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.716440 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe17509-a317-4eae-8edd-de1383398615-kube-api-access-4t9n4" pod "2fe17509-a317-4eae-8edd-de1383398615" (UID: "2fe17509-a317-4eae-8edd-de1383398615"). InnerVolumeSpecName "kube-api-access-4t9n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:19.716517 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.716485 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fe17509-a317-4eae-8edd-de1383398615-istio-podinfo" pod "2fe17509-a317-4eae-8edd-de1383398615" (UID: "2fe17509-a317-4eae-8edd-de1383398615"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:19.716719 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.716703 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe17509-a317-4eae-8edd-de1383398615-istio-token" pod "2fe17509-a317-4eae-8edd-de1383398615" (UID: "2fe17509-a317-4eae-8edd-de1383398615"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:19.815932 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.815888 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fe17509-a317-4eae-8edd-de1383398615-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:19.815932 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.815931 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t9n4\" (UniqueName: \"kubernetes.io/projected/2fe17509-a317-4eae-8edd-de1383398615-kube-api-access-4t9n4\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:19.815932 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.815941 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fe17509-a317-4eae-8edd-de1383398615-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:19.815932 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.815950 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fe17509-a317-4eae-8edd-de1383398615-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:19.962346 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.962266 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr"] Oct 30 21:51:19.976178 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:19.976150 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842"] Oct 30 21:51:20.097046 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.097011 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:20.098454 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.098430 5801 generic.go:356] "Generic (PLEG): container finished" podID="59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b" containerID="39ecc5d7f9f6bb1230e29c4e604d282222c9a4c48b1fc03391cabc2cbb1c4bce" exitCode=0 Oct 30 21:51:20.098557 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.098506 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760beae1-9k5tp" event={"ID":"59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b","Type":"ContainerDied","Data":"39ecc5d7f9f6bb1230e29c4e604d282222c9a4c48b1fc03391cabc2cbb1c4bce"} Oct 30 21:51:20.100375 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.100351 5801 generic.go:356] "Generic (PLEG): container finished" podID="2fe17509-a317-4eae-8edd-de1383398615" containerID="c9fc4971acbfbd843a554bf516ba427b62331da91968c3ba02d505b5705d3f1b" exitCode=0 Oct 30 21:51:20.100475 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.100402 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" Oct 30 21:51:20.100475 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.100425 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" event={"ID":"2fe17509-a317-4eae-8edd-de1383398615","Type":"ContainerDied","Data":"c9fc4971acbfbd843a554bf516ba427b62331da91968c3ba02d505b5705d3f1b"} Oct 30 21:51:20.100475 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.100463 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft" event={"ID":"2fe17509-a317-4eae-8edd-de1383398615","Type":"ContainerDied","Data":"876021985e124669c917eb1267ada0d4f073a89a960a96be483f243ddb2c2be8"} Oct 30 21:51:20.100621 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.100487 5801 scope.go:122] "RemoveContainer" containerID="c9fc4971acbfbd843a554bf516ba427b62331da91968c3ba02d505b5705d3f1b" Oct 30 21:51:20.101999 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.101980 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl" Oct 30 21:51:20.102151 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.102013 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl" event={"ID":"7de2823f-1b5e-434e-987a-133d6f101aa6","Type":"ContainerDied","Data":"f236a7fc8361004617c9379c247df6fa94ec33d47aff77130d062f211980f298"} Oct 30 21:51:20.102151 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.102054 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f236a7fc8361004617c9379c247df6fa94ec33d47aff77130d062f211980f298" Oct 30 21:51:20.104209 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.103996 5801 generic.go:356] "Generic (PLEG): container finished" podID="9ed9e521-5edd-4d8c-886a-adc56eeb5637" containerID="1517550c5d3c80484620bbf73ecd70169c7210328d84c66729bdf39facb74269" exitCode=0 Oct 30 21:51:20.104209 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.104132 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg" event={"ID":"9ed9e521-5edd-4d8c-886a-adc56eeb5637","Type":"ContainerDied","Data":"1517550c5d3c80484620bbf73ecd70169c7210328d84c66729bdf39facb74269"} Oct 30 21:51:20.118153 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.118127 5801 scope.go:122] "RemoveContainer" containerID="990d404bc28815dc548b2efe8ed1b456d8f7196f81375af3e75dd80ac7e15524" Oct 30 21:51:20.128461 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.128444 5801 scope.go:122] "RemoveContainer" containerID="c9fc4971acbfbd843a554bf516ba427b62331da91968c3ba02d505b5705d3f1b" Oct 30 21:51:20.128736 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:20.128718 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9fc4971acbfbd843a554bf516ba427b62331da91968c3ba02d505b5705d3f1b\": container with ID starting with c9fc4971acbfbd843a554bf516ba427b62331da91968c3ba02d505b5705d3f1b not found: ID does not exist" containerID="c9fc4971acbfbd843a554bf516ba427b62331da91968c3ba02d505b5705d3f1b" Oct 30 21:51:20.128908 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.128743 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9fc4971acbfbd843a554bf516ba427b62331da91968c3ba02d505b5705d3f1b"} err="failed to get container status \"c9fc4971acbfbd843a554bf516ba427b62331da91968c3ba02d505b5705d3f1b\": rpc error: code = NotFound desc = could not find container \"c9fc4971acbfbd843a554bf516ba427b62331da91968c3ba02d505b5705d3f1b\": container with ID starting with c9fc4971acbfbd843a554bf516ba427b62331da91968c3ba02d505b5705d3f1b not found: ID does not exist" Oct 30 21:51:20.128908 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.128774 5801 scope.go:122] "RemoveContainer" containerID="990d404bc28815dc548b2efe8ed1b456d8f7196f81375af3e75dd80ac7e15524" Oct 30 21:51:20.129029 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:20.129004 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"990d404bc28815dc548b2efe8ed1b456d8f7196f81375af3e75dd80ac7e15524\": container with ID starting with 990d404bc28815dc548b2efe8ed1b456d8f7196f81375af3e75dd80ac7e15524 not found: ID does not exist" containerID="990d404bc28815dc548b2efe8ed1b456d8f7196f81375af3e75dd80ac7e15524" Oct 30 21:51:20.129079 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.129040 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"990d404bc28815dc548b2efe8ed1b456d8f7196f81375af3e75dd80ac7e15524"} err="failed to get container status \"990d404bc28815dc548b2efe8ed1b456d8f7196f81375af3e75dd80ac7e15524\": rpc error: code = NotFound desc = could not find container \"990d404bc28815dc548b2efe8ed1b456d8f7196f81375af3e75dd80ac7e15524\": container with ID starting with 990d404bc28815dc548b2efe8ed1b456d8f7196f81375af3e75dd80ac7e15524 not found: ID does not exist" Oct 30 21:51:20.163730 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.163703 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft"] Oct 30 21:51:20.171548 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.171522 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-dc3edf-openshift-default-6d45f4d957-s5sft"] Oct 30 21:51:20.375769 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.375721 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:20.476476 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.476438 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:20.728010 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.727963 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9"] Oct 30 21:51:20.728428 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.728231 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" podUID="239d3785-9a31-40ed-9e63-2d7e1265b5a2" containerName="istio-proxy" containerID="cri-o://271fffe223a2d2862e0fc018dc5681a54424333b8242d1a7a11ef7b22dcff634" gracePeriod=30 Oct 30 21:51:20.764441 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:20.764401 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:21.013126 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.013103 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:51:21.033880 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.033847 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-workload-socket\") pod \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " Oct 30 21:51:21.034006 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.033886 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istiod-ca-cert\") pod \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " Oct 30 21:51:21.034006 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.033970 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-workload-certs\") pod \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " Oct 30 21:51:21.034136 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.034070 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-token\" (UniqueName: \"kubernetes.io/projected/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-token\") pod \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " Oct 30 21:51:21.034136 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.034097 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-workload-socket" pod "3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e" (UID: "3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:21.034136 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.034111 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-kube-api-access-wcrqd\" (UniqueName: \"kubernetes.io/projected/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-kube-api-access-wcrqd\") pod \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " Oct 30 21:51:21.034300 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.034219 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-envoy\") pod \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " Oct 30 21:51:21.034300 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.034253 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-workload-certs" pod "3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e" (UID: "3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:21.034300 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.034284 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istiod-ca-cert" pod "3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e" (UID: "3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:21.034458 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.034328 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-podinfo\") pod \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " Oct 30 21:51:21.034458 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.034354 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-data\") pod \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " Oct 30 21:51:21.034458 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.034373 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-credential-socket\") pod \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\" (UID: \"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e\") " Oct 30 21:51:21.034654 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.034622 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-data" pod "3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e" (UID: "3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:21.034729 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.034668 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-credential-socket" pod "3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e" (UID: "3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:21.034991 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.034965 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:21.035117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.034996 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:21.035117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.035012 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:21.035117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.035026 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:21.035117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.035040 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:21.036332 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.036308 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-kube-api-access-wcrqd" pod "3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e" (UID: "3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e"). InnerVolumeSpecName "kube-api-access-wcrqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:21.036332 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.036324 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-envoy" pod "3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e" (UID: "3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:21.036469 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.036346 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-token" pod "3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e" (UID: "3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:21.036530 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.036516 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-podinfo" pod "3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e" (UID: "3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:21.114472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.114440 5801 generic.go:356] "Generic (PLEG): container finished" podID="3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e" containerID="2234d70e1ea909adb643e94f3f7348c408d4b02151880f068245e2d826f44fc2" exitCode=0 Oct 30 21:51:21.114645 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.114491 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" Oct 30 21:51:21.114645 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.114529 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" event={"ID":"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e","Type":"ContainerDied","Data":"2234d70e1ea909adb643e94f3f7348c408d4b02151880f068245e2d826f44fc2"} Oct 30 21:51:21.114645 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.114583 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz" event={"ID":"3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e","Type":"ContainerDied","Data":"da0ef261e8a27f44a6e8a5ed165cf5b8a6be825eb998172bbe7e14111dcf7bd9"} Oct 30 21:51:21.114645 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.114612 5801 scope.go:122] "RemoveContainer" containerID="2234d70e1ea909adb643e94f3f7348c408d4b02151880f068245e2d826f44fc2" Oct 30 21:51:21.127785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.127743 5801 scope.go:122] "RemoveContainer" containerID="cbab4eddbdbb21198e1d111ef24547a637bb3ca3648360505872d2cb402dffe9" Oct 30 21:51:21.137199 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.137133 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:21.137199 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.137165 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcrqd\" (UniqueName: \"kubernetes.io/projected/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-kube-api-access-wcrqd\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:21.137541 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.137477 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:21.137541 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.137506 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:21.142431 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.142345 5801 scope.go:122] "RemoveContainer" containerID="2234d70e1ea909adb643e94f3f7348c408d4b02151880f068245e2d826f44fc2" Oct 30 21:51:21.142790 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:21.142745 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2234d70e1ea909adb643e94f3f7348c408d4b02151880f068245e2d826f44fc2\": container with ID starting with 2234d70e1ea909adb643e94f3f7348c408d4b02151880f068245e2d826f44fc2 not found: ID does not exist" containerID="2234d70e1ea909adb643e94f3f7348c408d4b02151880f068245e2d826f44fc2" Oct 30 21:51:21.142893 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.142796 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2234d70e1ea909adb643e94f3f7348c408d4b02151880f068245e2d826f44fc2"} err="failed to get container status \"2234d70e1ea909adb643e94f3f7348c408d4b02151880f068245e2d826f44fc2\": rpc error: code = NotFound desc = could not find container \"2234d70e1ea909adb643e94f3f7348c408d4b02151880f068245e2d826f44fc2\": container with ID starting with 2234d70e1ea909adb643e94f3f7348c408d4b02151880f068245e2d826f44fc2 not found: ID does not exist" Oct 30 21:51:21.142893 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.142824 5801 scope.go:122] "RemoveContainer" containerID="cbab4eddbdbb21198e1d111ef24547a637bb3ca3648360505872d2cb402dffe9" Oct 30 21:51:21.143127 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:21.143105 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbab4eddbdbb21198e1d111ef24547a637bb3ca3648360505872d2cb402dffe9\": container with ID starting with cbab4eddbdbb21198e1d111ef24547a637bb3ca3648360505872d2cb402dffe9 not found: ID does not exist" containerID="cbab4eddbdbb21198e1d111ef24547a637bb3ca3648360505872d2cb402dffe9" Oct 30 21:51:21.143201 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.143132 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbab4eddbdbb21198e1d111ef24547a637bb3ca3648360505872d2cb402dffe9"} err="failed to get container status \"cbab4eddbdbb21198e1d111ef24547a637bb3ca3648360505872d2cb402dffe9\": rpc error: code = NotFound desc = could not find container \"cbab4eddbdbb21198e1d111ef24547a637bb3ca3648360505872d2cb402dffe9\": container with ID starting with cbab4eddbdbb21198e1d111ef24547a637bb3ca3648360505872d2cb402dffe9 not found: ID does not exist" Oct 30 21:51:21.174063 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.174028 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz"] Oct 30 21:51:21.191349 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.191302 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-aad4dd-openshift-default-dd7545f6f-cktdz"] Oct 30 21:51:21.232534 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.232504 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f"] Oct 30 21:51:21.262062 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.261830 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb"] Oct 30 21:51:21.308420 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.308398 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760beae1-9k5tp" Oct 30 21:51:21.327474 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.327433 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:21.328112 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.328096 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg" Oct 30 21:51:21.338820 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.338798 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ed9e521-5edd-4d8c-886a-adc56eeb5637-kube-api-access-gbmk5\" (UniqueName: \"kubernetes.io/projected/9ed9e521-5edd-4d8c-886a-adc56eeb5637-kube-api-access-gbmk5\") pod \"9ed9e521-5edd-4d8c-886a-adc56eeb5637\" (UID: \"9ed9e521-5edd-4d8c-886a-adc56eeb5637\") " Oct 30 21:51:21.338952 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.338850 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ed9e521-5edd-4d8c-886a-adc56eeb5637-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed9e521-5edd-4d8c-886a-adc56eeb5637-maas-api-openshift-ca\") pod \"9ed9e521-5edd-4d8c-886a-adc56eeb5637\" (UID: \"9ed9e521-5edd-4d8c-886a-adc56eeb5637\") " Oct 30 21:51:21.338952 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.338931 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b-maas-api-xks-ca\") pod \"59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b\" (UID: \"59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b\") " Oct 30 21:51:21.339078 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.338953 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b-kube-api-access-sxgcx\" (UniqueName: \"kubernetes.io/projected/59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b-kube-api-access-sxgcx\") pod \"59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b\" (UID: \"59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b\") " Oct 30 21:51:21.339078 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.338977 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ed9e521-5edd-4d8c-886a-adc56eeb5637-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed9e521-5edd-4d8c-886a-adc56eeb5637-maas-api-xks-ca\") pod \"9ed9e521-5edd-4d8c-886a-adc56eeb5637\" (UID: \"9ed9e521-5edd-4d8c-886a-adc56eeb5637\") " Oct 30 21:51:21.339078 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.339031 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b-maas-api-openshift-ca\") pod \"59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b\" (UID: \"59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b\") " Oct 30 21:51:21.339283 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.339254 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed9e521-5edd-4d8c-886a-adc56eeb5637-maas-api-openshift-ca" pod "9ed9e521-5edd-4d8c-886a-adc56eeb5637" (UID: "9ed9e521-5edd-4d8c-886a-adc56eeb5637"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:21.339676 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.339650 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed9e521-5edd-4d8c-886a-adc56eeb5637-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:21.339816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.339700 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b-maas-api-openshift-ca" pod "59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b" (UID: "59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:21.341799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.341741 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b-maas-api-xks-ca" pod "59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b" (UID: "59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:21.341942 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.341745 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed9e521-5edd-4d8c-886a-adc56eeb5637-kube-api-access-gbmk5" pod "9ed9e521-5edd-4d8c-886a-adc56eeb5637" (UID: "9ed9e521-5edd-4d8c-886a-adc56eeb5637"). InnerVolumeSpecName "kube-api-access-gbmk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:21.343358 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.343338 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b-kube-api-access-sxgcx" pod "59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b" (UID: "59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b"). InnerVolumeSpecName "kube-api-access-sxgcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:21.343963 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.343940 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ed9e521-5edd-4d8c-886a-adc56eeb5637-maas-api-xks-ca" pod "9ed9e521-5edd-4d8c-886a-adc56eeb5637" (UID: "9ed9e521-5edd-4d8c-886a-adc56eeb5637"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:21.434034 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.433994 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:21.441243 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.441211 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:21.441371 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.441248 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbmk5\" (UniqueName: \"kubernetes.io/projected/9ed9e521-5edd-4d8c-886a-adc56eeb5637-kube-api-access-gbmk5\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:21.441371 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.441265 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:21.441371 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.441280 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxgcx\" (UniqueName: \"kubernetes.io/projected/59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b-kube-api-access-sxgcx\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:21.441371 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.441295 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed9e521-5edd-4d8c-886a-adc56eeb5637-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:21.462473 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.462439 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5"] Oct 30 21:51:21.463665 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.463642 5801 cpu_manager.go:414] "containerMap: removing container" podUID="7de2823f-1b5e-434e-987a-133d6f101aa6" containerName="revoke-keys" Oct 30 21:51:21.463801 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.463666 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7de2823f-1b5e-434e-987a-133d6f101aa6" containerName="revoke-keys" Oct 30 21:51:21.463801 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.463689 5801 cpu_manager.go:414] "containerMap: removing container" podUID="3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e" containerName="istio-proxy" Oct 30 21:51:21.463801 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.463698 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e" containerName="istio-proxy" Oct 30 21:51:21.463801 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.463732 5801 cpu_manager.go:414] "containerMap: removing container" podUID="2fe17509-a317-4eae-8edd-de1383398615" containerName="istio-proxy" Oct 30 21:51:21.463801 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.463744 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe17509-a317-4eae-8edd-de1383398615" containerName="istio-proxy" Oct 30 21:51:21.463801 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.463794 5801 cpu_manager.go:414] "containerMap: removing container" podUID="9ed9e521-5edd-4d8c-886a-adc56eeb5637" containerName="revoke-keys" Oct 30 21:51:21.463801 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.463802 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ed9e521-5edd-4d8c-886a-adc56eeb5637" containerName="revoke-keys" Oct 30 21:51:21.464172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.463836 5801 cpu_manager.go:414] "containerMap: removing container" podUID="c84c6d7e-7ce7-4caf-97ec-43267070cab3" containerName="revoke-keys" Oct 30 21:51:21.464172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.463845 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c84c6d7e-7ce7-4caf-97ec-43267070cab3" containerName="revoke-keys" Oct 30 21:51:21.464172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.463900 5801 cpu_manager.go:414] "containerMap: removing container" podUID="59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b" containerName="revoke-keys" Oct 30 21:51:21.464172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.463908 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b" containerName="revoke-keys" Oct 30 21:51:21.464172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.464158 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="c84c6d7e-7ce7-4caf-97ec-43267070cab3" containerName="revoke-keys" Oct 30 21:51:21.464416 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.464180 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe17509-a317-4eae-8edd-de1383398615" containerName="istio-proxy" Oct 30 21:51:21.464416 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.464197 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b" containerName="revoke-keys" Oct 30 21:51:21.464416 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.464215 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e" containerName="istio-proxy" Oct 30 21:51:21.464416 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.464232 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="7de2823f-1b5e-434e-987a-133d6f101aa6" containerName="revoke-keys" Oct 30 21:51:21.464416 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.464244 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e" containerName="istio-proxy" Oct 30 21:51:21.464416 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.464331 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ed9e521-5edd-4d8c-886a-adc56eeb5637" containerName="revoke-keys" Oct 30 21:51:21.474184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.474154 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5"] Oct 30 21:51:21.474298 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.474276 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5" Oct 30 21:51:21.526174 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.526086 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fe17509-a317-4eae-8edd-de1383398615" path="/var/lib/kubelet/pods/2fe17509-a317-4eae-8edd-de1383398615/volumes" Oct 30 21:51:21.526867 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.526846 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e" path="/var/lib/kubelet/pods/3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e/volumes" Oct 30 21:51:21.542220 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.542176 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24r6s\" (UniqueName: \"kubernetes.io/projected/5f421340-1852-46a4-9e66-4884e7c631f4-kube-api-access-24r6s\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5\" (UID: \"5f421340-1852-46a4-9e66-4884e7c631f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5" Oct 30 21:51:21.542422 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.542388 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f421340-1852-46a4-9e66-4884e7c631f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5\" (UID: \"5f421340-1852-46a4-9e66-4884e7c631f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5" Oct 30 21:51:21.542562 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.542536 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f421340-1852-46a4-9e66-4884e7c631f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5\" (UID: \"5f421340-1852-46a4-9e66-4884e7c631f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5" Oct 30 21:51:21.546926 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.546904 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:21.643405 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.643365 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f421340-1852-46a4-9e66-4884e7c631f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5\" (UID: \"5f421340-1852-46a4-9e66-4884e7c631f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5" Oct 30 21:51:21.643591 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.643436 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24r6s\" (UniqueName: \"kubernetes.io/projected/5f421340-1852-46a4-9e66-4884e7c631f4-kube-api-access-24r6s\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5\" (UID: \"5f421340-1852-46a4-9e66-4884e7c631f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5" Oct 30 21:51:21.643591 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.643573 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f421340-1852-46a4-9e66-4884e7c631f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5\" (UID: \"5f421340-1852-46a4-9e66-4884e7c631f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5" Oct 30 21:51:21.644263 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.644237 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f421340-1852-46a4-9e66-4884e7c631f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5\" (UID: \"5f421340-1852-46a4-9e66-4884e7c631f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5" Oct 30 21:51:21.645862 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.645834 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f421340-1852-46a4-9e66-4884e7c631f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5\" (UID: \"5f421340-1852-46a4-9e66-4884e7c631f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5" Oct 30 21:51:21.655936 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.655915 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24r6s\" (UniqueName: \"kubernetes.io/projected/5f421340-1852-46a4-9e66-4884e7c631f4-kube-api-access-24r6s\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5\" (UID: \"5f421340-1852-46a4-9e66-4884e7c631f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5" Oct 30 21:51:21.795299 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:21.795220 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5" Oct 30 21:51:22.078460 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:22.078416 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5"] Oct 30 21:51:22.079521 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:51:22.079497 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f421340_1852_46a4_9e66_4884e7c631f4.slice/crio-563b4fffe5df8c75bfc24a3ff22f96b8c1e303e3de44398ee1322a8ee75bdf55 WatchSource:0}: Error finding container 563b4fffe5df8c75bfc24a3ff22f96b8c1e303e3de44398ee1322a8ee75bdf55: Status 404 returned error can't find the container with id 563b4fffe5df8c75bfc24a3ff22f96b8c1e303e3de44398ee1322a8ee75bdf55 Oct 30 21:51:22.122805 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:22.122775 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5" event={"ID":"5f421340-1852-46a4-9e66-4884e7c631f4","Type":"ContainerStarted","Data":"563b4fffe5df8c75bfc24a3ff22f96b8c1e303e3de44398ee1322a8ee75bdf55"} Oct 30 21:51:22.124287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:22.124264 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760beae1-9k5tp" Oct 30 21:51:22.124403 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:22.124284 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760beae1-9k5tp" event={"ID":"59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b","Type":"ContainerDied","Data":"71e6aa4279b3962a6ac136ced5bc979c87f90e1d1be14ce0910586ca6c8dd91b"} Oct 30 21:51:22.124403 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:22.124326 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71e6aa4279b3962a6ac136ced5bc979c87f90e1d1be14ce0910586ca6c8dd91b" Oct 30 21:51:22.126740 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:22.126716 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg" event={"ID":"9ed9e521-5edd-4d8c-886a-adc56eeb5637","Type":"ContainerDied","Data":"17857516fd943e1f66e6ddb6dfcb6f66dd1871f055870a05ab24d79885b041b1"} Oct 30 21:51:22.126740 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:22.126770 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17857516fd943e1f66e6ddb6dfcb6f66dd1871f055870a05ab24d79885b041b1" Oct 30 21:51:22.126941 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:22.126798 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg" Oct 30 21:51:22.532593 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:22.532561 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:23.135976 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:23.135941 5801 generic.go:356] "Generic (PLEG): container finished" podID="5f421340-1852-46a4-9e66-4884e7c631f4" containerID="5c1960554ad5f9cffdd0b2b1fc609f317ebe581eab035dd1a2cd82d6b84b92fa" exitCode=0 Oct 30 21:51:23.136343 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:23.136028 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5" event={"ID":"5f421340-1852-46a4-9e66-4884e7c631f4","Type":"ContainerDied","Data":"5c1960554ad5f9cffdd0b2b1fc609f317ebe581eab035dd1a2cd82d6b84b92fa"} Oct 30 21:51:23.345034 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:23.344939 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:24.039544 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.039499 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:24.212578 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.212538 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" containerID="cri-o://2e2aff92c8b972908623a2676ced1c83afdc70f56a69d179fefbe42c90b4464a" gracePeriod=45 Oct 30 21:51:24.343402 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.343377 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5" Oct 30 21:51:24.377069 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.377041 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f421340-1852-46a4-9e66-4884e7c631f4-kube-api-access-24r6s\" (UniqueName: \"kubernetes.io/projected/5f421340-1852-46a4-9e66-4884e7c631f4-kube-api-access-24r6s\") pod \"5f421340-1852-46a4-9e66-4884e7c631f4\" (UID: \"5f421340-1852-46a4-9e66-4884e7c631f4\") " Oct 30 21:51:24.377216 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.377113 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f421340-1852-46a4-9e66-4884e7c631f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f421340-1852-46a4-9e66-4884e7c631f4-maas-api-openshift-ca\") pod \"5f421340-1852-46a4-9e66-4884e7c631f4\" (UID: \"5f421340-1852-46a4-9e66-4884e7c631f4\") " Oct 30 21:51:24.377275 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.377228 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f421340-1852-46a4-9e66-4884e7c631f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f421340-1852-46a4-9e66-4884e7c631f4-maas-api-xks-ca\") pod \"5f421340-1852-46a4-9e66-4884e7c631f4\" (UID: \"5f421340-1852-46a4-9e66-4884e7c631f4\") " Oct 30 21:51:24.377735 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.377698 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f421340-1852-46a4-9e66-4884e7c631f4-maas-api-openshift-ca" pod "5f421340-1852-46a4-9e66-4884e7c631f4" (UID: "5f421340-1852-46a4-9e66-4884e7c631f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:24.385614 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.385586 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f421340-1852-46a4-9e66-4884e7c631f4-maas-api-xks-ca" pod "5f421340-1852-46a4-9e66-4884e7c631f4" (UID: "5f421340-1852-46a4-9e66-4884e7c631f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:24.392912 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.392885 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f421340-1852-46a4-9e66-4884e7c631f4-kube-api-access-24r6s" pod "5f421340-1852-46a4-9e66-4884e7c631f4" (UID: "5f421340-1852-46a4-9e66-4884e7c631f4"). InnerVolumeSpecName "kube-api-access-24r6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:24.478559 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.478530 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f421340-1852-46a4-9e66-4884e7c631f4-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.478559 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.478557 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24r6s\" (UniqueName: \"kubernetes.io/projected/5f421340-1852-46a4-9e66-4884e7c631f4-kube-api-access-24r6s\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.478717 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.478569 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f421340-1852-46a4-9e66-4884e7c631f4-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.480801 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.480747 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:51:24.579473 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.579391 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-home\" (UniqueName: \"kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-home\") pod \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\" (UID: \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\") " Oct 30 21:51:24.579661 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.579520 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e33e49d7-3ed1-4ec8-ab79-de53674ea214-tls-certs\" (UniqueName: \"kubernetes.io/secret/e33e49d7-3ed1-4ec8-ab79-de53674ea214-tls-certs\") pod \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\" (UID: \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\") " Oct 30 21:51:24.579661 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.579572 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-model-cache\") pod \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\" (UID: \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\") " Oct 30 21:51:24.579661 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.579644 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-home" pod "e33e49d7-3ed1-4ec8-ab79-de53674ea214" (UID: "e33e49d7-3ed1-4ec8-ab79-de53674ea214"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:24.579911 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.579705 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-dshm\") pod \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\" (UID: \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\") " Oct 30 21:51:24.579911 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.579894 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-tmp-dir\") pod \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\" (UID: \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\") " Oct 30 21:51:24.580033 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.579951 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e33e49d7-3ed1-4ec8-ab79-de53674ea214-kube-api-access-btdbc\" (UniqueName: \"kubernetes.io/projected/e33e49d7-3ed1-4ec8-ab79-de53674ea214-kube-api-access-btdbc\") pod \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\" (UID: \"e33e49d7-3ed1-4ec8-ab79-de53674ea214\") " Oct 30 21:51:24.580033 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.579958 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-model-cache" pod "e33e49d7-3ed1-4ec8-ab79-de53674ea214" (UID: "e33e49d7-3ed1-4ec8-ab79-de53674ea214"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:24.580196 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.580178 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-tmp-dir" pod "e33e49d7-3ed1-4ec8-ab79-de53674ea214" (UID: "e33e49d7-3ed1-4ec8-ab79-de53674ea214"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:24.580517 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.580493 5801 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.580618 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.580522 5801 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.580618 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.580537 5801 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.582499 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.582468 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33e49d7-3ed1-4ec8-ab79-de53674ea214-kube-api-access-btdbc" pod "e33e49d7-3ed1-4ec8-ab79-de53674ea214" (UID: "e33e49d7-3ed1-4ec8-ab79-de53674ea214"). InnerVolumeSpecName "kube-api-access-btdbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:24.582598 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.582558 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e33e49d7-3ed1-4ec8-ab79-de53674ea214-tls-certs" pod "e33e49d7-3ed1-4ec8-ab79-de53674ea214" (UID: "e33e49d7-3ed1-4ec8-ab79-de53674ea214"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:24.582598 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.582580 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-dshm" pod "e33e49d7-3ed1-4ec8-ab79-de53674ea214" (UID: "e33e49d7-3ed1-4ec8-ab79-de53674ea214"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:24.682287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.682255 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btdbc\" (UniqueName: \"kubernetes.io/projected/e33e49d7-3ed1-4ec8-ab79-de53674ea214-kube-api-access-btdbc\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.682287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.682284 5801 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33e49d7-3ed1-4ec8-ab79-de53674ea214-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.682287 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.682295 5801 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33e49d7-3ed1-4ec8-ab79-de53674ea214-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:24.924809 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:24.924690 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:25.155550 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.155524 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5" Oct 30 21:51:25.155742 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.155561 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5" event={"ID":"5f421340-1852-46a4-9e66-4884e7c631f4","Type":"ContainerDied","Data":"563b4fffe5df8c75bfc24a3ff22f96b8c1e303e3de44398ee1322a8ee75bdf55"} Oct 30 21:51:25.155742 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.155609 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="563b4fffe5df8c75bfc24a3ff22f96b8c1e303e3de44398ee1322a8ee75bdf55" Oct 30 21:51:25.157217 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.157189 5801 generic.go:356] "Generic (PLEG): container finished" podID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerID="2e2aff92c8b972908623a2676ced1c83afdc70f56a69d179fefbe42c90b4464a" exitCode=0 Oct 30 21:51:25.157217 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.157236 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" event={"ID":"e33e49d7-3ed1-4ec8-ab79-de53674ea214","Type":"ContainerDied","Data":"2e2aff92c8b972908623a2676ced1c83afdc70f56a69d179fefbe42c90b4464a"} Oct 30 21:51:25.157485 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.157266 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" event={"ID":"e33e49d7-3ed1-4ec8-ab79-de53674ea214","Type":"ContainerDied","Data":"e3f0d8ce003ba1df04f96efde0d854c606a8e6e58c458c1fb7e4989434b57072"} Oct 30 21:51:25.157485 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.157264 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298" Oct 30 21:51:25.157485 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.157338 5801 scope.go:122] "RemoveContainer" containerID="2e2aff92c8b972908623a2676ced1c83afdc70f56a69d179fefbe42c90b4464a" Oct 30 21:51:25.170262 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.170240 5801 scope.go:122] "RemoveContainer" containerID="2e2aff92c8b972908623a2676ced1c83afdc70f56a69d179fefbe42c90b4464a" Oct 30 21:51:25.170540 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:25.170512 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2aff92c8b972908623a2676ced1c83afdc70f56a69d179fefbe42c90b4464a\": container with ID starting with 2e2aff92c8b972908623a2676ced1c83afdc70f56a69d179fefbe42c90b4464a not found: ID does not exist" containerID="2e2aff92c8b972908623a2676ced1c83afdc70f56a69d179fefbe42c90b4464a" Oct 30 21:51:25.170607 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.170547 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2aff92c8b972908623a2676ced1c83afdc70f56a69d179fefbe42c90b4464a"} err="failed to get container status \"2e2aff92c8b972908623a2676ced1c83afdc70f56a69d179fefbe42c90b4464a\": rpc error: code = NotFound desc = could not find container \"2e2aff92c8b972908623a2676ced1c83afdc70f56a69d179fefbe42c90b4464a\": container with ID starting with 2e2aff92c8b972908623a2676ced1c83afdc70f56a69d179fefbe42c90b4464a not found: ID does not exist" Oct 30 21:51:25.184657 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.184600 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298"] Oct 30 21:51:25.187685 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.187666 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-db0b54/distinct-2-w2-db0b54-kserve-6cd6f9f957-p7298"] Oct 30 21:51:25.207149 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.207124 5801 patch_prober.go:29] interesting pod/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:51:25.207224 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.207175 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" podUID="239d3785-9a31-40ed-9e63-2d7e1265b5a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:51:25.293456 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.293430 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw"] Oct 30 21:51:25.362246 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.362213 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:25.466909 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.466824 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:25.526960 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.526921 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" path="/var/lib/kubelet/pods/e33e49d7-3ed1-4ec8-ab79-de53674ea214/volumes" Oct 30 21:51:25.527742 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.527438 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-473ccdb9-fsbfd"] Oct 30 21:51:25.529734 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.529001 5801 cpu_manager.go:414] "containerMap: removing container" podUID="3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e" containerName="istio-proxy" Oct 30 21:51:25.529734 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.529027 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cb1fb25-0a54-4f2e-9bcb-88e2ebcaef2e" containerName="istio-proxy" Oct 30 21:51:25.529734 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.529052 5801 cpu_manager.go:414] "containerMap: removing container" podUID="2fe17509-a317-4eae-8edd-de1383398615" containerName="istio-proxy" Oct 30 21:51:25.529734 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.529061 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe17509-a317-4eae-8edd-de1383398615" containerName="istio-proxy" Oct 30 21:51:25.529734 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.529072 5801 cpu_manager.go:414] "containerMap: removing container" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" Oct 30 21:51:25.529734 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.529079 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" Oct 30 21:51:25.529734 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.529122 5801 cpu_manager.go:414] "containerMap: removing container" podUID="5f421340-1852-46a4-9e66-4884e7c631f4" containerName="revoke-keys" Oct 30 21:51:25.529734 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.529130 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f421340-1852-46a4-9e66-4884e7c631f4" containerName="revoke-keys" Oct 30 21:51:25.529734 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.529283 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f421340-1852-46a4-9e66-4884e7c631f4" containerName="revoke-keys" Oct 30 21:51:25.529734 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.529306 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="e33e49d7-3ed1-4ec8-ab79-de53674ea214" containerName="main" Oct 30 21:51:25.529734 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.529321 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe17509-a317-4eae-8edd-de1383398615" containerName="istio-proxy" Oct 30 21:51:25.534504 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.534485 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f664bea-rpgtd"] Oct 30 21:51:25.534664 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.534645 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-473ccdb9-fsbfd" Oct 30 21:51:25.539981 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.539243 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:51:25.543059 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.540221 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:51:25.543059 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.540294 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:51:25.543059 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.540415 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:51:25.546990 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.546965 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-473ccdb9-fsbfd"] Oct 30 21:51:25.547099 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.547010 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f664bea-rpgtd"] Oct 30 21:51:25.547099 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.547096 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f664bea-rpgtd" Oct 30 21:51:25.602785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.593798 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb9944d-e392-451e-89a6-1d6030441138-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f664bea-rpgtd\" (UID: \"efb9944d-e392-451e-89a6-1d6030441138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f664bea-rpgtd" Oct 30 21:51:25.602785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.593928 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-473ccdb9-fsbfd\" (UID: \"e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473ccdb9-fsbfd" Oct 30 21:51:25.602785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.593963 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j267w\" (UniqueName: \"kubernetes.io/projected/e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606-kube-api-access-j267w\") pod \"maas-api-revoke-sub-473ccdb9-fsbfd\" (UID: \"e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473ccdb9-fsbfd" Oct 30 21:51:25.602785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.594004 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb9944d-e392-451e-89a6-1d6030441138-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f664bea-rpgtd\" (UID: \"efb9944d-e392-451e-89a6-1d6030441138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f664bea-rpgtd" Oct 30 21:51:25.602785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.594086 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606-maas-api-xks-ca\") pod \"maas-api-revoke-sub-473ccdb9-fsbfd\" (UID: \"e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473ccdb9-fsbfd" Oct 30 21:51:25.602785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.594144 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8b9l\" (UniqueName: \"kubernetes.io/projected/efb9944d-e392-451e-89a6-1d6030441138-kube-api-access-n8b9l\") pod \"maas-api-revoke-sub-8f664bea-rpgtd\" (UID: \"efb9944d-e392-451e-89a6-1d6030441138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f664bea-rpgtd" Oct 30 21:51:25.695454 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.695414 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb9944d-e392-451e-89a6-1d6030441138-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f664bea-rpgtd\" (UID: \"efb9944d-e392-451e-89a6-1d6030441138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f664bea-rpgtd" Oct 30 21:51:25.695661 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.695461 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-473ccdb9-fsbfd\" (UID: \"e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473ccdb9-fsbfd" Oct 30 21:51:25.695661 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.695487 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j267w\" (UniqueName: \"kubernetes.io/projected/e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606-kube-api-access-j267w\") pod \"maas-api-revoke-sub-473ccdb9-fsbfd\" (UID: \"e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473ccdb9-fsbfd" Oct 30 21:51:25.695661 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.695521 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb9944d-e392-451e-89a6-1d6030441138-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f664bea-rpgtd\" (UID: \"efb9944d-e392-451e-89a6-1d6030441138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f664bea-rpgtd" Oct 30 21:51:25.695661 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.695595 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606-maas-api-xks-ca\") pod \"maas-api-revoke-sub-473ccdb9-fsbfd\" (UID: \"e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473ccdb9-fsbfd" Oct 30 21:51:25.695973 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.695682 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8b9l\" (UniqueName: \"kubernetes.io/projected/efb9944d-e392-451e-89a6-1d6030441138-kube-api-access-n8b9l\") pod \"maas-api-revoke-sub-8f664bea-rpgtd\" (UID: \"efb9944d-e392-451e-89a6-1d6030441138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f664bea-rpgtd" Oct 30 21:51:25.696381 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.696355 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-473ccdb9-fsbfd\" (UID: \"e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473ccdb9-fsbfd" Oct 30 21:51:25.696501 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.696460 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb9944d-e392-451e-89a6-1d6030441138-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f664bea-rpgtd\" (UID: \"efb9944d-e392-451e-89a6-1d6030441138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f664bea-rpgtd" Oct 30 21:51:25.698056 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.698035 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606-maas-api-xks-ca\") pod \"maas-api-revoke-sub-473ccdb9-fsbfd\" (UID: \"e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473ccdb9-fsbfd" Oct 30 21:51:25.698153 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.698041 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb9944d-e392-451e-89a6-1d6030441138-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f664bea-rpgtd\" (UID: \"efb9944d-e392-451e-89a6-1d6030441138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f664bea-rpgtd" Oct 30 21:51:25.708348 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.708322 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j267w\" (UniqueName: \"kubernetes.io/projected/e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606-kube-api-access-j267w\") pod \"maas-api-revoke-sub-473ccdb9-fsbfd\" (UID: \"e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473ccdb9-fsbfd" Oct 30 21:51:25.708798 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.708745 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8b9l\" (UniqueName: \"kubernetes.io/projected/efb9944d-e392-451e-89a6-1d6030441138-kube-api-access-n8b9l\") pod \"maas-api-revoke-sub-8f664bea-rpgtd\" (UID: \"efb9944d-e392-451e-89a6-1d6030441138\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f664bea-rpgtd" Oct 30 21:51:25.857962 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.857932 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-473ccdb9-fsbfd" Oct 30 21:51:25.866140 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.866095 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:25.888621 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:25.887588 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f664bea-rpgtd" Oct 30 21:51:26.026872 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.026848 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:51:26.106620 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.106589 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-credential-socket\") pod \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " Oct 30 21:51:26.106790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.106640 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-token\" (UniqueName: \"kubernetes.io/projected/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-token\") pod \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " Oct 30 21:51:26.106790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.106685 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istiod-ca-cert\") pod \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " Oct 30 21:51:26.106790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.106738 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-podinfo\") pod \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " Oct 30 21:51:26.106944 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.106799 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-workload-certs\") pod \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " Oct 30 21:51:26.106944 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.106851 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-data\") pod \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " Oct 30 21:51:26.106944 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.106906 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-envoy\") pod \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " Oct 30 21:51:26.107100 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.106901 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-credential-socket" pod "239d3785-9a31-40ed-9e63-2d7e1265b5a2" (UID: "239d3785-9a31-40ed-9e63-2d7e1265b5a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:26.107100 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.107058 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-workload-socket\") pod \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " Oct 30 21:51:26.107216 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.107154 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/239d3785-9a31-40ed-9e63-2d7e1265b5a2-kube-api-access-2lxcm\" (UniqueName: \"kubernetes.io/projected/239d3785-9a31-40ed-9e63-2d7e1265b5a2-kube-api-access-2lxcm\") pod \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\" (UID: \"239d3785-9a31-40ed-9e63-2d7e1265b5a2\") " Oct 30 21:51:26.107216 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.107194 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-workload-certs" pod "239d3785-9a31-40ed-9e63-2d7e1265b5a2" (UID: "239d3785-9a31-40ed-9e63-2d7e1265b5a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:26.107329 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.107228 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istiod-ca-cert" pod "239d3785-9a31-40ed-9e63-2d7e1265b5a2" (UID: "239d3785-9a31-40ed-9e63-2d7e1265b5a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:26.107389 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.107327 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-workload-socket" pod "239d3785-9a31-40ed-9e63-2d7e1265b5a2" (UID: "239d3785-9a31-40ed-9e63-2d7e1265b5a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:26.107830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.107800 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:26.107830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.107821 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-data" pod "239d3785-9a31-40ed-9e63-2d7e1265b5a2" (UID: "239d3785-9a31-40ed-9e63-2d7e1265b5a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:26.108032 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.107832 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:26.108032 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.107866 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:26.108032 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.107884 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:26.109051 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.109025 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-envoy" pod "239d3785-9a31-40ed-9e63-2d7e1265b5a2" (UID: "239d3785-9a31-40ed-9e63-2d7e1265b5a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:26.109179 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.109052 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-podinfo" pod "239d3785-9a31-40ed-9e63-2d7e1265b5a2" (UID: "239d3785-9a31-40ed-9e63-2d7e1265b5a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:26.109179 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.109070 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-token" pod "239d3785-9a31-40ed-9e63-2d7e1265b5a2" (UID: "239d3785-9a31-40ed-9e63-2d7e1265b5a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:26.109179 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.109157 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/239d3785-9a31-40ed-9e63-2d7e1265b5a2-kube-api-access-2lxcm" pod "239d3785-9a31-40ed-9e63-2d7e1265b5a2" (UID: "239d3785-9a31-40ed-9e63-2d7e1265b5a2"). InnerVolumeSpecName "kube-api-access-2lxcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:26.168296 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.168262 5801 generic.go:356] "Generic (PLEG): container finished" podID="239d3785-9a31-40ed-9e63-2d7e1265b5a2" containerID="271fffe223a2d2862e0fc018dc5681a54424333b8242d1a7a11ef7b22dcff634" exitCode=0 Oct 30 21:51:26.168422 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.168318 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" Oct 30 21:51:26.168422 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.168340 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" event={"ID":"239d3785-9a31-40ed-9e63-2d7e1265b5a2","Type":"ContainerDied","Data":"271fffe223a2d2862e0fc018dc5681a54424333b8242d1a7a11ef7b22dcff634"} Oct 30 21:51:26.168422 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.168396 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9" event={"ID":"239d3785-9a31-40ed-9e63-2d7e1265b5a2","Type":"ContainerDied","Data":"307f28457d4c802b56902776fe37616caf91a01ad0521b8e8de9448c326f9db6"} Oct 30 21:51:26.168422 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.168415 5801 scope.go:122] "RemoveContainer" containerID="271fffe223a2d2862e0fc018dc5681a54424333b8242d1a7a11ef7b22dcff634" Oct 30 21:51:26.180195 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.180175 5801 scope.go:122] "RemoveContainer" containerID="6c3d90cae6558521bde9d8ddb74aec810038ed575c9b634e214e3158f58a9763" Oct 30 21:51:26.181797 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:51:26.181768 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ae1bc2_03cd_4df3_bf3d_f04ad66e3606.slice/crio-9415395a8772a60ee43e632907bee4f80d7b10cafdbf7b2b9c780299550d710a WatchSource:0}: Error finding container 9415395a8772a60ee43e632907bee4f80d7b10cafdbf7b2b9c780299550d710a: Status 404 returned error can't find the container with id 9415395a8772a60ee43e632907bee4f80d7b10cafdbf7b2b9c780299550d710a Oct 30 21:51:26.183132 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.183107 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-473ccdb9-fsbfd"] Oct 30 21:51:26.190359 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.190070 5801 scope.go:122] "RemoveContainer" containerID="271fffe223a2d2862e0fc018dc5681a54424333b8242d1a7a11ef7b22dcff634" Oct 30 21:51:26.190359 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:26.190309 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"271fffe223a2d2862e0fc018dc5681a54424333b8242d1a7a11ef7b22dcff634\": container with ID starting with 271fffe223a2d2862e0fc018dc5681a54424333b8242d1a7a11ef7b22dcff634 not found: ID does not exist" containerID="271fffe223a2d2862e0fc018dc5681a54424333b8242d1a7a11ef7b22dcff634" Oct 30 21:51:26.190359 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.190331 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"271fffe223a2d2862e0fc018dc5681a54424333b8242d1a7a11ef7b22dcff634"} err="failed to get container status \"271fffe223a2d2862e0fc018dc5681a54424333b8242d1a7a11ef7b22dcff634\": rpc error: code = NotFound desc = could not find container \"271fffe223a2d2862e0fc018dc5681a54424333b8242d1a7a11ef7b22dcff634\": container with ID starting with 271fffe223a2d2862e0fc018dc5681a54424333b8242d1a7a11ef7b22dcff634 not found: ID does not exist" Oct 30 21:51:26.190359 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.190350 5801 scope.go:122] "RemoveContainer" containerID="6c3d90cae6558521bde9d8ddb74aec810038ed575c9b634e214e3158f58a9763" Oct 30 21:51:26.190633 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:26.190615 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c3d90cae6558521bde9d8ddb74aec810038ed575c9b634e214e3158f58a9763\": container with ID starting with 6c3d90cae6558521bde9d8ddb74aec810038ed575c9b634e214e3158f58a9763 not found: ID does not exist" containerID="6c3d90cae6558521bde9d8ddb74aec810038ed575c9b634e214e3158f58a9763" Oct 30 21:51:26.190679 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.190637 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3d90cae6558521bde9d8ddb74aec810038ed575c9b634e214e3158f58a9763"} err="failed to get container status \"6c3d90cae6558521bde9d8ddb74aec810038ed575c9b634e214e3158f58a9763\": rpc error: code = NotFound desc = could not find container \"6c3d90cae6558521bde9d8ddb74aec810038ed575c9b634e214e3158f58a9763\": container with ID starting with 6c3d90cae6558521bde9d8ddb74aec810038ed575c9b634e214e3158f58a9763 not found: ID does not exist" Oct 30 21:51:26.196098 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.196074 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9"] Oct 30 21:51:26.201073 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.201052 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f608f7-openshift-default-68678c9b85-2kst9"] Oct 30 21:51:26.208921 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.208900 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lxcm\" (UniqueName: \"kubernetes.io/projected/239d3785-9a31-40ed-9e63-2d7e1265b5a2-kube-api-access-2lxcm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:26.208987 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.208925 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:26.208987 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.208935 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:26.208987 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.208944 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:26.208987 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.208954 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239d3785-9a31-40ed-9e63-2d7e1265b5a2-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:26.230544 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.230512 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f664bea-rpgtd"] Oct 30 21:51:26.231409 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:51:26.231388 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb9944d_e392_451e_89a6_1d6030441138.slice/crio-fc9967de7d9c2bfab58e2863ee1aa344152eaaca01a781d95a8c3f2e35ce30a6 WatchSource:0}: Error finding container fc9967de7d9c2bfab58e2863ee1aa344152eaaca01a781d95a8c3f2e35ce30a6: Status 404 returned error can't find the container with id fc9967de7d9c2bfab58e2863ee1aa344152eaaca01a781d95a8c3f2e35ce30a6 Oct 30 21:51:26.360591 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.360564 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq"] Oct 30 21:51:26.371177 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.371146 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5"] Oct 30 21:51:26.451791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.451740 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:26.641533 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.641496 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:26.739452 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:26.739419 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:27.180595 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:27.180567 5801 generic.go:356] "Generic (PLEG): container finished" podID="efb9944d-e392-451e-89a6-1d6030441138" containerID="c36aa2da8f3a2296342436c6e00da945763e08711432479f089464787a076aac" exitCode=0 Oct 30 21:51:27.180742 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:27.180646 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f664bea-rpgtd" event={"ID":"efb9944d-e392-451e-89a6-1d6030441138","Type":"ContainerDied","Data":"c36aa2da8f3a2296342436c6e00da945763e08711432479f089464787a076aac"} Oct 30 21:51:27.180742 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:27.180697 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f664bea-rpgtd" event={"ID":"efb9944d-e392-451e-89a6-1d6030441138","Type":"ContainerStarted","Data":"fc9967de7d9c2bfab58e2863ee1aa344152eaaca01a781d95a8c3f2e35ce30a6"} Oct 30 21:51:27.182390 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:27.182365 5801 generic.go:356] "Generic (PLEG): container finished" podID="e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606" containerID="5ed01d4ddca6b34be16ee90b1865dd9d952a74d9badb564252f824d4770a4ca2" exitCode=0 Oct 30 21:51:27.182481 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:27.182442 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-473ccdb9-fsbfd" event={"ID":"e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606","Type":"ContainerDied","Data":"5ed01d4ddca6b34be16ee90b1865dd9d952a74d9badb564252f824d4770a4ca2"} Oct 30 21:51:27.182530 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:27.182483 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-473ccdb9-fsbfd" event={"ID":"e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606","Type":"ContainerStarted","Data":"9415395a8772a60ee43e632907bee4f80d7b10cafdbf7b2b9c780299550d710a"} Oct 30 21:51:27.456382 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:27.456290 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:27.527871 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:27.527837 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="239d3785-9a31-40ed-9e63-2d7e1265b5a2" path="/var/lib/kubelet/pods/239d3785-9a31-40ed-9e63-2d7e1265b5a2/volumes" Oct 30 21:51:27.600628 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:27.600592 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k"] Oct 30 21:51:27.609200 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:27.609173 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5"] Oct 30 21:51:27.916502 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:27.916465 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:28.014650 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.014613 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:28.268790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.267426 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9374298-5ln8q"] Oct 30 21:51:28.268790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.268291 5801 cpu_manager.go:414] "containerMap: removing container" podUID="239d3785-9a31-40ed-9e63-2d7e1265b5a2" containerName="istio-proxy" Oct 30 21:51:28.268790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.268313 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="239d3785-9a31-40ed-9e63-2d7e1265b5a2" containerName="istio-proxy" Oct 30 21:51:28.268790 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.268486 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="239d3785-9a31-40ed-9e63-2d7e1265b5a2" containerName="istio-proxy" Oct 30 21:51:28.277895 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.275424 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9374298-5ln8q" Oct 30 21:51:28.304552 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.302723 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9374298-5ln8q"] Oct 30 21:51:28.343947 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.342032 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44f975b9-767d-4a1c-87bd-e8043cdf84c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9374298-5ln8q\" (UID: \"44f975b9-767d-4a1c-87bd-e8043cdf84c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9374298-5ln8q" Oct 30 21:51:28.343947 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.342118 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44f975b9-767d-4a1c-87bd-e8043cdf84c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9374298-5ln8q\" (UID: \"44f975b9-767d-4a1c-87bd-e8043cdf84c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9374298-5ln8q" Oct 30 21:51:28.343947 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.342148 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsx59\" (UniqueName: \"kubernetes.io/projected/44f975b9-767d-4a1c-87bd-e8043cdf84c0-kube-api-access-hsx59\") pod \"maas-api-revoke-sub-f9374298-5ln8q\" (UID: \"44f975b9-767d-4a1c-87bd-e8043cdf84c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9374298-5ln8q" Oct 30 21:51:28.383405 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.383376 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f664bea-rpgtd" Oct 30 21:51:28.419125 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.419093 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-473ccdb9-fsbfd" Oct 30 21:51:28.444254 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.444175 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efb9944d-e392-451e-89a6-1d6030441138-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb9944d-e392-451e-89a6-1d6030441138-maas-api-openshift-ca\") pod \"efb9944d-e392-451e-89a6-1d6030441138\" (UID: \"efb9944d-e392-451e-89a6-1d6030441138\") " Oct 30 21:51:28.444511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.444354 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606-maas-api-openshift-ca\") pod \"e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606\" (UID: \"e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606\") " Oct 30 21:51:28.444511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.444400 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efb9944d-e392-451e-89a6-1d6030441138-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb9944d-e392-451e-89a6-1d6030441138-maas-api-xks-ca\") pod \"efb9944d-e392-451e-89a6-1d6030441138\" (UID: \"efb9944d-e392-451e-89a6-1d6030441138\") " Oct 30 21:51:28.444511 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.444448 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb9944d-e392-451e-89a6-1d6030441138-kube-api-access-n8b9l\" (UniqueName: \"kubernetes.io/projected/efb9944d-e392-451e-89a6-1d6030441138-kube-api-access-n8b9l\") pod \"efb9944d-e392-451e-89a6-1d6030441138\" (UID: \"efb9944d-e392-451e-89a6-1d6030441138\") " Oct 30 21:51:28.444691 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.444507 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606-maas-api-xks-ca\") pod \"e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606\" (UID: \"e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606\") " Oct 30 21:51:28.444691 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.444574 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606-kube-api-access-j267w\" (UniqueName: \"kubernetes.io/projected/e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606-kube-api-access-j267w\") pod \"e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606\" (UID: \"e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606\") " Oct 30 21:51:28.445094 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.445067 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44f975b9-767d-4a1c-87bd-e8043cdf84c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9374298-5ln8q\" (UID: \"44f975b9-767d-4a1c-87bd-e8043cdf84c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9374298-5ln8q" Oct 30 21:51:28.445223 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.445147 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44f975b9-767d-4a1c-87bd-e8043cdf84c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9374298-5ln8q\" (UID: \"44f975b9-767d-4a1c-87bd-e8043cdf84c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9374298-5ln8q" Oct 30 21:51:28.445223 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.445173 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsx59\" (UniqueName: \"kubernetes.io/projected/44f975b9-767d-4a1c-87bd-e8043cdf84c0-kube-api-access-hsx59\") pod \"maas-api-revoke-sub-f9374298-5ln8q\" (UID: \"44f975b9-767d-4a1c-87bd-e8043cdf84c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9374298-5ln8q" Oct 30 21:51:28.445876 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.445838 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb9944d-e392-451e-89a6-1d6030441138-maas-api-openshift-ca" pod "efb9944d-e392-451e-89a6-1d6030441138" (UID: "efb9944d-e392-451e-89a6-1d6030441138"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:28.446162 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.446143 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606-maas-api-openshift-ca" pod "e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606" (UID: "e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:28.449211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.448615 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb9944d-e392-451e-89a6-1d6030441138-maas-api-xks-ca" pod "efb9944d-e392-451e-89a6-1d6030441138" (UID: "efb9944d-e392-451e-89a6-1d6030441138"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:28.451990 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.451962 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44f975b9-767d-4a1c-87bd-e8043cdf84c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9374298-5ln8q\" (UID: \"44f975b9-767d-4a1c-87bd-e8043cdf84c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9374298-5ln8q" Oct 30 21:51:28.454028 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.453985 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44f975b9-767d-4a1c-87bd-e8043cdf84c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9374298-5ln8q\" (UID: \"44f975b9-767d-4a1c-87bd-e8043cdf84c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9374298-5ln8q" Oct 30 21:51:28.457659 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.457616 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsx59\" (UniqueName: \"kubernetes.io/projected/44f975b9-767d-4a1c-87bd-e8043cdf84c0-kube-api-access-hsx59\") pod \"maas-api-revoke-sub-f9374298-5ln8q\" (UID: \"44f975b9-767d-4a1c-87bd-e8043cdf84c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9374298-5ln8q" Oct 30 21:51:28.458881 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.458844 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606-kube-api-access-j267w" pod "e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606" (UID: "e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606"). InnerVolumeSpecName "kube-api-access-j267w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:28.458881 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.458837 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606-maas-api-xks-ca" pod "e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606" (UID: "e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:28.459008 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.458876 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb9944d-e392-451e-89a6-1d6030441138-kube-api-access-n8b9l" pod "efb9944d-e392-451e-89a6-1d6030441138" (UID: "efb9944d-e392-451e-89a6-1d6030441138"). InnerVolumeSpecName "kube-api-access-n8b9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:28.546411 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.546380 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:28.546411 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.546408 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb9944d-e392-451e-89a6-1d6030441138-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:28.546411 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.546419 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8b9l\" (UniqueName: \"kubernetes.io/projected/efb9944d-e392-451e-89a6-1d6030441138-kube-api-access-n8b9l\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:28.546649 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.546429 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:28.546649 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.546438 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j267w\" (UniqueName: \"kubernetes.io/projected/e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606-kube-api-access-j267w\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:28.546649 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.546446 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb9944d-e392-451e-89a6-1d6030441138-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:28.555615 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.555583 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:28.642250 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.642212 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9374298-5ln8q" Oct 30 21:51:28.928283 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:28.928254 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9374298-5ln8q"] Oct 30 21:51:28.928841 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:51:28.928817 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f975b9_767d_4a1c_87bd_e8043cdf84c0.slice/crio-d8d6dce8a759580dc613b6cd5406df8c493fcb479c900808134a1c209d317396 WatchSource:0}: Error finding container d8d6dce8a759580dc613b6cd5406df8c493fcb479c900808134a1c209d317396: Status 404 returned error can't find the container with id d8d6dce8a759580dc613b6cd5406df8c493fcb479c900808134a1c209d317396 Oct 30 21:51:29.199328 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:29.199302 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f664bea-rpgtd" Oct 30 21:51:29.199472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:29.199321 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f664bea-rpgtd" event={"ID":"efb9944d-e392-451e-89a6-1d6030441138","Type":"ContainerDied","Data":"fc9967de7d9c2bfab58e2863ee1aa344152eaaca01a781d95a8c3f2e35ce30a6"} Oct 30 21:51:29.199472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:29.199374 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc9967de7d9c2bfab58e2863ee1aa344152eaaca01a781d95a8c3f2e35ce30a6" Oct 30 21:51:29.200704 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:29.200681 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9374298-5ln8q" event={"ID":"44f975b9-767d-4a1c-87bd-e8043cdf84c0","Type":"ContainerStarted","Data":"d8d6dce8a759580dc613b6cd5406df8c493fcb479c900808134a1c209d317396"} Oct 30 21:51:29.202146 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:29.202126 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-473ccdb9-fsbfd" Oct 30 21:51:29.202230 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:29.202190 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-473ccdb9-fsbfd" event={"ID":"e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606","Type":"ContainerDied","Data":"9415395a8772a60ee43e632907bee4f80d7b10cafdbf7b2b9c780299550d710a"} Oct 30 21:51:29.202230 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:29.202220 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9415395a8772a60ee43e632907bee4f80d7b10cafdbf7b2b9c780299550d710a" Oct 30 21:51:29.251949 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:29.251893 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:29.982122 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:29.982090 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:30.210698 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:30.210661 5801 generic.go:356] "Generic (PLEG): container finished" podID="44f975b9-767d-4a1c-87bd-e8043cdf84c0" containerID="9c7c960f9b06f5949df7098b1efa99f0248a8571dcf3af5b613d82546996d5df" exitCode=0 Oct 30 21:51:30.210878 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:30.210696 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9374298-5ln8q" event={"ID":"44f975b9-767d-4a1c-87bd-e8043cdf84c0","Type":"ContainerDied","Data":"9c7c960f9b06f5949df7098b1efa99f0248a8571dcf3af5b613d82546996d5df"} Oct 30 21:51:30.280138 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:30.280104 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:51:30.294708 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:30.294677 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw" podUID="abc6bc4f-b6fe-49aa-9f79-41da164ffa08" containerName="payload-processing" containerID="cri-o://14157862f3c04e54eb7387cdfb2527aaf664321bcb8403c105ce4962fd1d228d" gracePeriod=25 Oct 30 21:51:30.535849 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:30.535769 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:30.671241 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:30.671196 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:30.737632 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:30.737602 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg"] Oct 30 21:51:30.740733 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:30.740712 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0c62ab-fr4xg"] Oct 30 21:51:30.770031 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:30.770002 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:31.023964 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:31.023931 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:31.124290 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:31.124253 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:31.164103 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:31.164075 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5"] Oct 30 21:51:31.167366 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:31.167340 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db0b54-smxn5"] Oct 30 21:51:31.253993 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:31.253935 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:31.353218 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:31.353179 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:31.390685 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:31.390661 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9374298-5ln8q" Oct 30 21:51:31.483536 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:31.483508 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44f975b9-767d-4a1c-87bd-e8043cdf84c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44f975b9-767d-4a1c-87bd-e8043cdf84c0-maas-api-xks-ca\") pod \"44f975b9-767d-4a1c-87bd-e8043cdf84c0\" (UID: \"44f975b9-767d-4a1c-87bd-e8043cdf84c0\") " Oct 30 21:51:31.483725 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:31.483630 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44f975b9-767d-4a1c-87bd-e8043cdf84c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44f975b9-767d-4a1c-87bd-e8043cdf84c0-maas-api-openshift-ca\") pod \"44f975b9-767d-4a1c-87bd-e8043cdf84c0\" (UID: \"44f975b9-767d-4a1c-87bd-e8043cdf84c0\") " Oct 30 21:51:31.483824 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:31.483763 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f975b9-767d-4a1c-87bd-e8043cdf84c0-kube-api-access-hsx59\" (UniqueName: \"kubernetes.io/projected/44f975b9-767d-4a1c-87bd-e8043cdf84c0-kube-api-access-hsx59\") pod \"44f975b9-767d-4a1c-87bd-e8043cdf84c0\" (UID: \"44f975b9-767d-4a1c-87bd-e8043cdf84c0\") " Oct 30 21:51:31.484063 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:31.484035 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f975b9-767d-4a1c-87bd-e8043cdf84c0-maas-api-openshift-ca" pod "44f975b9-767d-4a1c-87bd-e8043cdf84c0" (UID: "44f975b9-767d-4a1c-87bd-e8043cdf84c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:31.484169 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:31.484153 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44f975b9-767d-4a1c-87bd-e8043cdf84c0-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:31.485722 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:31.485699 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44f975b9-767d-4a1c-87bd-e8043cdf84c0-maas-api-xks-ca" pod "44f975b9-767d-4a1c-87bd-e8043cdf84c0" (UID: "44f975b9-767d-4a1c-87bd-e8043cdf84c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:31.485722 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:31.485707 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f975b9-767d-4a1c-87bd-e8043cdf84c0-kube-api-access-hsx59" pod "44f975b9-767d-4a1c-87bd-e8043cdf84c0" (UID: "44f975b9-767d-4a1c-87bd-e8043cdf84c0"). InnerVolumeSpecName "kube-api-access-hsx59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:31.523858 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:31.523831 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87025c3e-8014-4283-8f26-32822e095d93" path="/var/lib/kubelet/pods/87025c3e-8014-4283-8f26-32822e095d93/volumes" Oct 30 21:51:31.524302 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:31.524288 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ed9e521-5edd-4d8c-886a-adc56eeb5637" path="/var/lib/kubelet/pods/9ed9e521-5edd-4d8c-886a-adc56eeb5637/volumes" Oct 30 21:51:31.585714 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:31.585677 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsx59\" (UniqueName: \"kubernetes.io/projected/44f975b9-767d-4a1c-87bd-e8043cdf84c0-kube-api-access-hsx59\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:31.585714 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:31.585710 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44f975b9-767d-4a1c-87bd-e8043cdf84c0-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:31.690586 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:31.690550 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:31.789852 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:31.789748 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:32.227990 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:32.227959 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9374298-5ln8q" event={"ID":"44f975b9-767d-4a1c-87bd-e8043cdf84c0","Type":"ContainerDied","Data":"d8d6dce8a759580dc613b6cd5406df8c493fcb479c900808134a1c209d317396"} Oct 30 21:51:32.227990 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:32.227975 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9374298-5ln8q" Oct 30 21:51:32.228437 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:32.228006 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8d6dce8a759580dc613b6cd5406df8c493fcb479c900808134a1c209d317396" Oct 30 21:51:32.380901 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:32.380869 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:32.601851 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:32.601783 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k" podUID="35404e22-3c01-4e3c-bcf6-960df3dadfe0" containerName="payload-processing" containerID="cri-o://0647854103e1e7c22bd59ce00da9b50ca2024503bcfc69f3c5b3e28ff2cda9d2" gracePeriod=25 Oct 30 21:51:32.610223 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:32.610193 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5" podUID="00c8f036-7f24-4c36-94c4-a09a666338a2" containerName="payload-pre-processing" containerID="cri-o://9ca757fbec5d607fde6a87feb184c7f4843f463daf594fa70df1d11c55b31328" gracePeriod=25 Oct 30 21:51:32.866476 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:32.866408 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k" Oct 30 21:51:32.901614 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:32.901570 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35404e22-3c01-4e3c-bcf6-960df3dadfe0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35404e22-3c01-4e3c-bcf6-960df3dadfe0-plugins-config-volume\") pod \"35404e22-3c01-4e3c-bcf6-960df3dadfe0\" (UID: \"35404e22-3c01-4e3c-bcf6-960df3dadfe0\") " Oct 30 21:51:32.901823 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:32.901661 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35404e22-3c01-4e3c-bcf6-960df3dadfe0-kube-api-access-2g5qv\" (UniqueName: \"kubernetes.io/projected/35404e22-3c01-4e3c-bcf6-960df3dadfe0-kube-api-access-2g5qv\") pod \"35404e22-3c01-4e3c-bcf6-960df3dadfe0\" (UID: \"35404e22-3c01-4e3c-bcf6-960df3dadfe0\") " Oct 30 21:51:32.902693 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:32.902667 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35404e22-3c01-4e3c-bcf6-960df3dadfe0-plugins-config-volume" pod "35404e22-3c01-4e3c-bcf6-960df3dadfe0" (UID: "35404e22-3c01-4e3c-bcf6-960df3dadfe0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:32.904384 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:32.904355 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35404e22-3c01-4e3c-bcf6-960df3dadfe0-kube-api-access-2g5qv" pod "35404e22-3c01-4e3c-bcf6-960df3dadfe0" (UID: "35404e22-3c01-4e3c-bcf6-960df3dadfe0"). InnerVolumeSpecName "kube-api-access-2g5qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:32.938022 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:32.937998 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5" Oct 30 21:51:33.003245 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.003212 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c8f036-7f24-4c36-94c4-a09a666338a2-kube-api-access-kb967\" (UniqueName: \"kubernetes.io/projected/00c8f036-7f24-4c36-94c4-a09a666338a2-kube-api-access-kb967\") pod \"00c8f036-7f24-4c36-94c4-a09a666338a2\" (UID: \"00c8f036-7f24-4c36-94c4-a09a666338a2\") " Oct 30 21:51:33.003392 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.003296 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c8f036-7f24-4c36-94c4-a09a666338a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c8f036-7f24-4c36-94c4-a09a666338a2-plugins-config-volume\") pod \"00c8f036-7f24-4c36-94c4-a09a666338a2\" (UID: \"00c8f036-7f24-4c36-94c4-a09a666338a2\") " Oct 30 21:51:33.003650 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.003623 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c8f036-7f24-4c36-94c4-a09a666338a2-plugins-config-volume" pod "00c8f036-7f24-4c36-94c4-a09a666338a2" (UID: "00c8f036-7f24-4c36-94c4-a09a666338a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:33.004100 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.004077 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c8f036-7f24-4c36-94c4-a09a666338a2-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:33.004171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.004108 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35404e22-3c01-4e3c-bcf6-960df3dadfe0-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:33.004171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.004125 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g5qv\" (UniqueName: \"kubernetes.io/projected/35404e22-3c01-4e3c-bcf6-960df3dadfe0-kube-api-access-2g5qv\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:33.005412 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.005390 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c8f036-7f24-4c36-94c4-a09a666338a2-kube-api-access-kb967" pod "00c8f036-7f24-4c36-94c4-a09a666338a2" (UID: "00c8f036-7f24-4c36-94c4-a09a666338a2"). InnerVolumeSpecName "kube-api-access-kb967". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:33.031979 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.031954 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3c67196-c2mnf"] Oct 30 21:51:33.032459 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.032444 5801 cpu_manager.go:414] "containerMap: removing container" podUID="239d3785-9a31-40ed-9e63-2d7e1265b5a2" containerName="istio-proxy" Oct 30 21:51:33.032518 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.032460 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="239d3785-9a31-40ed-9e63-2d7e1265b5a2" containerName="istio-proxy" Oct 30 21:51:33.032518 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.032495 5801 cpu_manager.go:414] "containerMap: removing container" podUID="44f975b9-767d-4a1c-87bd-e8043cdf84c0" containerName="revoke-keys" Oct 30 21:51:33.032518 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.032500 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44f975b9-767d-4a1c-87bd-e8043cdf84c0" containerName="revoke-keys" Oct 30 21:51:33.032518 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.032508 5801 cpu_manager.go:414] "containerMap: removing container" podUID="e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606" containerName="revoke-keys" Oct 30 21:51:33.032518 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.032514 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606" containerName="revoke-keys" Oct 30 21:51:33.032666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.032523 5801 cpu_manager.go:414] "containerMap: removing container" podUID="35404e22-3c01-4e3c-bcf6-960df3dadfe0" containerName="payload-processing" Oct 30 21:51:33.032666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.032528 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35404e22-3c01-4e3c-bcf6-960df3dadfe0" containerName="payload-processing" Oct 30 21:51:33.032666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.032537 5801 cpu_manager.go:414] "containerMap: removing container" podUID="00c8f036-7f24-4c36-94c4-a09a666338a2" containerName="payload-pre-processing" Oct 30 21:51:33.032666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.032541 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c8f036-7f24-4c36-94c4-a09a666338a2" containerName="payload-pre-processing" Oct 30 21:51:33.032666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.032549 5801 cpu_manager.go:414] "containerMap: removing container" podUID="efb9944d-e392-451e-89a6-1d6030441138" containerName="revoke-keys" Oct 30 21:51:33.032666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.032554 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb9944d-e392-451e-89a6-1d6030441138" containerName="revoke-keys" Oct 30 21:51:33.032666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.032618 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="239d3785-9a31-40ed-9e63-2d7e1265b5a2" containerName="istio-proxy" Oct 30 21:51:33.032666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.032630 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="44f975b9-767d-4a1c-87bd-e8043cdf84c0" containerName="revoke-keys" Oct 30 21:51:33.032666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.032636 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606" containerName="revoke-keys" Oct 30 21:51:33.032666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.032642 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="35404e22-3c01-4e3c-bcf6-960df3dadfe0" containerName="payload-processing" Oct 30 21:51:33.032666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.032648 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c8f036-7f24-4c36-94c4-a09a666338a2" containerName="payload-pre-processing" Oct 30 21:51:33.032666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.032655 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb9944d-e392-451e-89a6-1d6030441138" containerName="revoke-keys" Oct 30 21:51:33.035973 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.035959 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-212c9eb0-9fb99"] Oct 30 21:51:33.036089 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.036072 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c67196-c2mnf" Oct 30 21:51:33.039490 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.039476 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c9eb0-9fb99" Oct 30 21:51:33.042783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.042744 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:51:33.045673 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.045581 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3c67196-c2mnf"] Oct 30 21:51:33.046798 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.045915 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:51:33.050241 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.050221 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-212c9eb0-9fb99"] Oct 30 21:51:33.104822 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.104793 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9e4efc8-551c-4352-a036-030ba1117dc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-212c9eb0-9fb99\" (UID: \"b9e4efc8-551c-4352-a036-030ba1117dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c9eb0-9fb99" Oct 30 21:51:33.104822 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.104824 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb9l5\" (UniqueName: \"kubernetes.io/projected/b9e4efc8-551c-4352-a036-030ba1117dc3-kube-api-access-rb9l5\") pod \"maas-api-revoke-sub-212c9eb0-9fb99\" (UID: \"b9e4efc8-551c-4352-a036-030ba1117dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c9eb0-9fb99" Oct 30 21:51:33.105014 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.104867 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a8a85ae-4b64-473a-beac-cbde16b4ea4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3c67196-c2mnf\" (UID: \"6a8a85ae-4b64-473a-beac-cbde16b4ea4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c67196-c2mnf" Oct 30 21:51:33.105014 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.104968 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h5j4\" (UniqueName: \"kubernetes.io/projected/6a8a85ae-4b64-473a-beac-cbde16b4ea4f-kube-api-access-6h5j4\") pod \"maas-api-revoke-sub-c3c67196-c2mnf\" (UID: \"6a8a85ae-4b64-473a-beac-cbde16b4ea4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c67196-c2mnf" Oct 30 21:51:33.105137 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.105074 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9e4efc8-551c-4352-a036-030ba1117dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-212c9eb0-9fb99\" (UID: \"b9e4efc8-551c-4352-a036-030ba1117dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c9eb0-9fb99" Oct 30 21:51:33.105197 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.105148 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a8a85ae-4b64-473a-beac-cbde16b4ea4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3c67196-c2mnf\" (UID: \"6a8a85ae-4b64-473a-beac-cbde16b4ea4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c67196-c2mnf" Oct 30 21:51:33.105312 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.105298 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb967\" (UniqueName: \"kubernetes.io/projected/00c8f036-7f24-4c36-94c4-a09a666338a2-kube-api-access-kb967\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:33.122565 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.122493 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:51:33.122680 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.122600 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:51:33.206868 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.206842 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9e4efc8-551c-4352-a036-030ba1117dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-212c9eb0-9fb99\" (UID: \"b9e4efc8-551c-4352-a036-030ba1117dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c9eb0-9fb99" Oct 30 21:51:33.207018 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.206898 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a8a85ae-4b64-473a-beac-cbde16b4ea4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3c67196-c2mnf\" (UID: \"6a8a85ae-4b64-473a-beac-cbde16b4ea4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c67196-c2mnf" Oct 30 21:51:33.207018 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.206976 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9e4efc8-551c-4352-a036-030ba1117dc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-212c9eb0-9fb99\" (UID: \"b9e4efc8-551c-4352-a036-030ba1117dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c9eb0-9fb99" Oct 30 21:51:33.207153 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.207086 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb9l5\" (UniqueName: \"kubernetes.io/projected/b9e4efc8-551c-4352-a036-030ba1117dc3-kube-api-access-rb9l5\") pod \"maas-api-revoke-sub-212c9eb0-9fb99\" (UID: \"b9e4efc8-551c-4352-a036-030ba1117dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c9eb0-9fb99" Oct 30 21:51:33.207219 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.207157 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a8a85ae-4b64-473a-beac-cbde16b4ea4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3c67196-c2mnf\" (UID: \"6a8a85ae-4b64-473a-beac-cbde16b4ea4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c67196-c2mnf" Oct 30 21:51:33.207219 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.207188 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h5j4\" (UniqueName: \"kubernetes.io/projected/6a8a85ae-4b64-473a-beac-cbde16b4ea4f-kube-api-access-6h5j4\") pod \"maas-api-revoke-sub-c3c67196-c2mnf\" (UID: \"6a8a85ae-4b64-473a-beac-cbde16b4ea4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c67196-c2mnf" Oct 30 21:51:33.207608 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.207580 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9e4efc8-551c-4352-a036-030ba1117dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-212c9eb0-9fb99\" (UID: \"b9e4efc8-551c-4352-a036-030ba1117dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c9eb0-9fb99" Oct 30 21:51:33.207840 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.207819 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a8a85ae-4b64-473a-beac-cbde16b4ea4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3c67196-c2mnf\" (UID: \"6a8a85ae-4b64-473a-beac-cbde16b4ea4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c67196-c2mnf" Oct 30 21:51:33.209195 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.209174 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a8a85ae-4b64-473a-beac-cbde16b4ea4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3c67196-c2mnf\" (UID: \"6a8a85ae-4b64-473a-beac-cbde16b4ea4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c67196-c2mnf" Oct 30 21:51:33.209286 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.209242 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9e4efc8-551c-4352-a036-030ba1117dc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-212c9eb0-9fb99\" (UID: \"b9e4efc8-551c-4352-a036-030ba1117dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c9eb0-9fb99" Oct 30 21:51:33.219607 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.219587 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb9l5\" (UniqueName: \"kubernetes.io/projected/b9e4efc8-551c-4352-a036-030ba1117dc3-kube-api-access-rb9l5\") pod \"maas-api-revoke-sub-212c9eb0-9fb99\" (UID: \"b9e4efc8-551c-4352-a036-030ba1117dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c9eb0-9fb99" Oct 30 21:51:33.219860 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.219845 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h5j4\" (UniqueName: \"kubernetes.io/projected/6a8a85ae-4b64-473a-beac-cbde16b4ea4f-kube-api-access-6h5j4\") pod \"maas-api-revoke-sub-c3c67196-c2mnf\" (UID: \"6a8a85ae-4b64-473a-beac-cbde16b4ea4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c67196-c2mnf" Oct 30 21:51:33.236682 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.236654 5801 generic.go:356] "Generic (PLEG): container finished" podID="00c8f036-7f24-4c36-94c4-a09a666338a2" containerID="9ca757fbec5d607fde6a87feb184c7f4843f463daf594fa70df1d11c55b31328" exitCode=0 Oct 30 21:51:33.237048 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.236715 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5" Oct 30 21:51:33.237048 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.236733 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5" event={"ID":"00c8f036-7f24-4c36-94c4-a09a666338a2","Type":"ContainerDied","Data":"9ca757fbec5d607fde6a87feb184c7f4843f463daf594fa70df1d11c55b31328"} Oct 30 21:51:33.237048 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.236823 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5" event={"ID":"00c8f036-7f24-4c36-94c4-a09a666338a2","Type":"ContainerDied","Data":"9dc573a00ce9df95a49757a2e6f491b41ceb7d7198b051e42dad0fc3a7369a22"} Oct 30 21:51:33.237048 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.236848 5801 scope.go:122] "RemoveContainer" containerID="9ca757fbec5d607fde6a87feb184c7f4843f463daf594fa70df1d11c55b31328" Oct 30 21:51:33.238430 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.238407 5801 generic.go:356] "Generic (PLEG): container finished" podID="35404e22-3c01-4e3c-bcf6-960df3dadfe0" containerID="0647854103e1e7c22bd59ce00da9b50ca2024503bcfc69f3c5b3e28ff2cda9d2" exitCode=0 Oct 30 21:51:33.238535 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.238437 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k" event={"ID":"35404e22-3c01-4e3c-bcf6-960df3dadfe0","Type":"ContainerDied","Data":"0647854103e1e7c22bd59ce00da9b50ca2024503bcfc69f3c5b3e28ff2cda9d2"} Oct 30 21:51:33.238535 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.238468 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k" Oct 30 21:51:33.238535 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.238475 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k" event={"ID":"35404e22-3c01-4e3c-bcf6-960df3dadfe0","Type":"ContainerDied","Data":"647499fe09b13260c69863f6c27c647d2945ccda6c8d7be8b95fd63ba54bf2b5"} Oct 30 21:51:33.248963 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.248947 5801 scope.go:122] "RemoveContainer" containerID="9ca757fbec5d607fde6a87feb184c7f4843f463daf594fa70df1d11c55b31328" Oct 30 21:51:33.249240 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:33.249222 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ca757fbec5d607fde6a87feb184c7f4843f463daf594fa70df1d11c55b31328\": container with ID starting with 9ca757fbec5d607fde6a87feb184c7f4843f463daf594fa70df1d11c55b31328 not found: ID does not exist" containerID="9ca757fbec5d607fde6a87feb184c7f4843f463daf594fa70df1d11c55b31328" Oct 30 21:51:33.249299 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.249247 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca757fbec5d607fde6a87feb184c7f4843f463daf594fa70df1d11c55b31328"} err="failed to get container status \"9ca757fbec5d607fde6a87feb184c7f4843f463daf594fa70df1d11c55b31328\": rpc error: code = NotFound desc = could not find container \"9ca757fbec5d607fde6a87feb184c7f4843f463daf594fa70df1d11c55b31328\": container with ID starting with 9ca757fbec5d607fde6a87feb184c7f4843f463daf594fa70df1d11c55b31328 not found: ID does not exist" Oct 30 21:51:33.249299 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.249263 5801 scope.go:122] "RemoveContainer" containerID="0647854103e1e7c22bd59ce00da9b50ca2024503bcfc69f3c5b3e28ff2cda9d2" Oct 30 21:51:33.258934 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.258875 5801 scope.go:122] "RemoveContainer" containerID="0647854103e1e7c22bd59ce00da9b50ca2024503bcfc69f3c5b3e28ff2cda9d2" Oct 30 21:51:33.259572 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:33.259451 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0647854103e1e7c22bd59ce00da9b50ca2024503bcfc69f3c5b3e28ff2cda9d2\": container with ID starting with 0647854103e1e7c22bd59ce00da9b50ca2024503bcfc69f3c5b3e28ff2cda9d2 not found: ID does not exist" containerID="0647854103e1e7c22bd59ce00da9b50ca2024503bcfc69f3c5b3e28ff2cda9d2" Oct 30 21:51:33.259572 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.259501 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0647854103e1e7c22bd59ce00da9b50ca2024503bcfc69f3c5b3e28ff2cda9d2"} err="failed to get container status \"0647854103e1e7c22bd59ce00da9b50ca2024503bcfc69f3c5b3e28ff2cda9d2\": rpc error: code = NotFound desc = could not find container \"0647854103e1e7c22bd59ce00da9b50ca2024503bcfc69f3c5b3e28ff2cda9d2\": container with ID starting with 0647854103e1e7c22bd59ce00da9b50ca2024503bcfc69f3c5b3e28ff2cda9d2 not found: ID does not exist" Oct 30 21:51:33.263681 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.263660 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5"] Oct 30 21:51:33.266863 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.266843 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0c62ab-697dbbccff-wnjk5"] Oct 30 21:51:33.277037 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.277017 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k"] Oct 30 21:51:33.281610 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.281586 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0c62ab-67b8fdb565-r926k"] Oct 30 21:51:33.354982 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.354949 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c67196-c2mnf" Oct 30 21:51:33.358857 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.358823 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:33.373271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.373218 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c9eb0-9fb99" Oct 30 21:51:33.526103 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.526069 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c8f036-7f24-4c36-94c4-a09a666338a2" path="/var/lib/kubelet/pods/00c8f036-7f24-4c36-94c4-a09a666338a2/volumes" Oct 30 21:51:33.526671 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.526647 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35404e22-3c01-4e3c-bcf6-960df3dadfe0" path="/var/lib/kubelet/pods/35404e22-3c01-4e3c-bcf6-960df3dadfe0/volumes" Oct 30 21:51:33.662783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.662742 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3c67196-c2mnf"] Oct 30 21:51:33.664080 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:51:33.664054 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a8a85ae_4b64_473a_beac_cbde16b4ea4f.slice/crio-6fdda8b7d9c52a024fea49d1fb976f148b2ca89063a52fd215f1d5a0b3ba0674 WatchSource:0}: Error finding container 6fdda8b7d9c52a024fea49d1fb976f148b2ca89063a52fd215f1d5a0b3ba0674: Status 404 returned error can't find the container with id 6fdda8b7d9c52a024fea49d1fb976f148b2ca89063a52fd215f1d5a0b3ba0674 Oct 30 21:51:33.712266 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.712237 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-212c9eb0-9fb99"] Oct 30 21:51:33.713021 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:51:33.712994 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e4efc8_551c_4352_a036_030ba1117dc3.slice/crio-e75239999530ad56cd60d11acabd001b2067ba5ef3d84bda5bd4aca99c6ef2b1 WatchSource:0}: Error finding container e75239999530ad56cd60d11acabd001b2067ba5ef3d84bda5bd4aca99c6ef2b1: Status 404 returned error can't find the container with id e75239999530ad56cd60d11acabd001b2067ba5ef3d84bda5bd4aca99c6ef2b1 Oct 30 21:51:33.971205 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:33.971176 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw" podUID="abc6bc4f-b6fe-49aa-9f79-41da164ffa08" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:51:34.147370 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:34.147328 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:34.248709 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:34.248615 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c9eb0-9fb99" event={"ID":"b9e4efc8-551c-4352-a036-030ba1117dc3","Type":"ContainerStarted","Data":"e2d93c13d490a6bb78c809cf7cc1f0b06828d16ca7f7862ab09113a7c340dde2"} Oct 30 21:51:34.248709 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:34.248681 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c9eb0-9fb99" event={"ID":"b9e4efc8-551c-4352-a036-030ba1117dc3","Type":"ContainerStarted","Data":"e75239999530ad56cd60d11acabd001b2067ba5ef3d84bda5bd4aca99c6ef2b1"} Oct 30 21:51:34.251411 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:34.251381 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c67196-c2mnf" event={"ID":"6a8a85ae-4b64-473a-beac-cbde16b4ea4f","Type":"ContainerStarted","Data":"97e8f497bfad441622b58dc9273a9bb41c8afd0a92f167d11300a4c5ac47a348"} Oct 30 21:51:34.251550 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:34.251427 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c67196-c2mnf" event={"ID":"6a8a85ae-4b64-473a-beac-cbde16b4ea4f","Type":"ContainerStarted","Data":"6fdda8b7d9c52a024fea49d1fb976f148b2ca89063a52fd215f1d5a0b3ba0674"} Oct 30 21:51:34.270350 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:34.270295 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c9eb0-9fb99" podStartSLOduration=1.270283465 podStartE2EDuration="1.270283465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:34.266921387 +0000 UTC m=+1362.862738064" watchObservedRunningTime="2026-10-30 21:51:34.270283465 +0000 UTC m=+1362.866100161" Oct 30 21:51:34.286965 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:34.286927 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c67196-c2mnf" podStartSLOduration=2.286915546 podStartE2EDuration="2.286915546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:34.285149313 +0000 UTC m=+1362.880965986" watchObservedRunningTime="2026-10-30 21:51:34.286915546 +0000 UTC m=+1362.882732255" Oct 30 21:51:34.954949 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:34.954861 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:34.983450 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:34.983410 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" containerID="cri-o://18c8ea1eabde5238d47b8c4447efbafa9ab1177f7a7cb1b04acf4832e17a12ff" gracePeriod=45 Oct 30 21:51:34.996527 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:34.996499 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" containerID="cri-o://290f91d4dfd4a1e83f4363e8306afeb90b1a4dd2916a01c86bfe4abc5f6dd870" gracePeriod=45 Oct 30 21:51:35.300470 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.293615 5801 generic.go:356] "Generic (PLEG): container finished" podID="6a8a85ae-4b64-473a-beac-cbde16b4ea4f" containerID="97e8f497bfad441622b58dc9273a9bb41c8afd0a92f167d11300a4c5ac47a348" exitCode=0 Oct 30 21:51:35.300470 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.293727 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c67196-c2mnf" event={"ID":"6a8a85ae-4b64-473a-beac-cbde16b4ea4f","Type":"ContainerDied","Data":"97e8f497bfad441622b58dc9273a9bb41c8afd0a92f167d11300a4c5ac47a348"} Oct 30 21:51:35.300470 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.296856 5801 generic.go:356] "Generic (PLEG): container finished" podID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerID="290f91d4dfd4a1e83f4363e8306afeb90b1a4dd2916a01c86bfe4abc5f6dd870" exitCode=0 Oct 30 21:51:35.300470 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.297025 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" event={"ID":"d40a938f-1aa7-4897-b5a7-91257132baf7","Type":"ContainerDied","Data":"290f91d4dfd4a1e83f4363e8306afeb90b1a4dd2916a01c86bfe4abc5f6dd870"} Oct 30 21:51:35.300470 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.300400 5801 generic.go:356] "Generic (PLEG): container finished" podID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerID="18c8ea1eabde5238d47b8c4447efbafa9ab1177f7a7cb1b04acf4832e17a12ff" exitCode=0 Oct 30 21:51:35.301154 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.300581 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" event={"ID":"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5","Type":"ContainerDied","Data":"18c8ea1eabde5238d47b8c4447efbafa9ab1177f7a7cb1b04acf4832e17a12ff"} Oct 30 21:51:35.302965 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.302936 5801 generic.go:356] "Generic (PLEG): container finished" podID="b9e4efc8-551c-4352-a036-030ba1117dc3" containerID="e2d93c13d490a6bb78c809cf7cc1f0b06828d16ca7f7862ab09113a7c340dde2" exitCode=0 Oct 30 21:51:35.303116 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.303042 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c9eb0-9fb99" event={"ID":"b9e4efc8-551c-4352-a036-030ba1117dc3","Type":"ContainerDied","Data":"e2d93c13d490a6bb78c809cf7cc1f0b06828d16ca7f7862ab09113a7c340dde2"} Oct 30 21:51:35.351828 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.351125 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:51:35.369461 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.369439 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:51:35.449665 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.449625 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-model-cache\") pod \"d40a938f-1aa7-4897-b5a7-91257132baf7\" (UID: \"d40a938f-1aa7-4897-b5a7-91257132baf7\") " Oct 30 21:51:35.449867 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.449678 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-kube-api-access-clk7h\" (UniqueName: \"kubernetes.io/projected/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-kube-api-access-clk7h\") pod \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\" (UID: \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\") " Oct 30 21:51:35.449867 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.449802 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-tmp-dir\") pod \"d40a938f-1aa7-4897-b5a7-91257132baf7\" (UID: \"d40a938f-1aa7-4897-b5a7-91257132baf7\") " Oct 30 21:51:35.449867 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.449863 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-home\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-home\") pod \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\" (UID: \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\") " Oct 30 21:51:35.450044 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.449924 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d40a938f-1aa7-4897-b5a7-91257132baf7-kube-api-access-6hrg6\" (UniqueName: \"kubernetes.io/projected/d40a938f-1aa7-4897-b5a7-91257132baf7-kube-api-access-6hrg6\") pod \"d40a938f-1aa7-4897-b5a7-91257132baf7\" (UID: \"d40a938f-1aa7-4897-b5a7-91257132baf7\") " Oct 30 21:51:35.450044 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.449957 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-tmp-dir\") pod \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\" (UID: \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\") " Oct 30 21:51:35.450044 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.449993 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-dshm\") pod \"d40a938f-1aa7-4897-b5a7-91257132baf7\" (UID: \"d40a938f-1aa7-4897-b5a7-91257132baf7\") " Oct 30 21:51:35.450190 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.450061 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-model-cache\") pod \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\" (UID: \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\") " Oct 30 21:51:35.450190 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.450067 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-model-cache" pod "d40a938f-1aa7-4897-b5a7-91257132baf7" (UID: "d40a938f-1aa7-4897-b5a7-91257132baf7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:35.450190 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.450122 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d40a938f-1aa7-4897-b5a7-91257132baf7-tls-certs\" (UniqueName: \"kubernetes.io/secret/d40a938f-1aa7-4897-b5a7-91257132baf7-tls-certs\") pod \"d40a938f-1aa7-4897-b5a7-91257132baf7\" (UID: \"d40a938f-1aa7-4897-b5a7-91257132baf7\") " Oct 30 21:51:35.450190 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.450176 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-tls-certs\") pod \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\" (UID: \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\") " Oct 30 21:51:35.450405 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.450227 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-dshm\") pod \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\" (UID: \"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5\") " Oct 30 21:51:35.450405 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.450278 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-home\" (UniqueName: \"kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-home\") pod \"d40a938f-1aa7-4897-b5a7-91257132baf7\" (UID: \"d40a938f-1aa7-4897-b5a7-91257132baf7\") " Oct 30 21:51:35.450881 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.450855 5801 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.450881 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.450277 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-home" pod "d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" (UID: "d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:35.451452 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.451421 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-home" pod "d40a938f-1aa7-4897-b5a7-91257132baf7" (UID: "d40a938f-1aa7-4897-b5a7-91257132baf7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:35.451554 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.451482 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-tmp-dir" pod "d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" (UID: "d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:35.451554 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.451521 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-model-cache" pod "d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" (UID: "d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:35.451681 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.451656 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-tmp-dir" pod "d40a938f-1aa7-4897-b5a7-91257132baf7" (UID: "d40a938f-1aa7-4897-b5a7-91257132baf7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:35.458190 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.457070 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-dshm" pod "d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" (UID: "d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:35.458190 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.457113 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-dshm" pod "d40a938f-1aa7-4897-b5a7-91257132baf7" (UID: "d40a938f-1aa7-4897-b5a7-91257132baf7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:35.458190 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.457115 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-kube-api-access-clk7h" pod "d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" (UID: "d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5"). InnerVolumeSpecName "kube-api-access-clk7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:35.458440 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.458306 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d40a938f-1aa7-4897-b5a7-91257132baf7-tls-certs" pod "d40a938f-1aa7-4897-b5a7-91257132baf7" (UID: "d40a938f-1aa7-4897-b5a7-91257132baf7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:35.458643 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.458534 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-tls-certs" pod "d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" (UID: "d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:35.458643 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.458622 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40a938f-1aa7-4897-b5a7-91257132baf7-kube-api-access-6hrg6" pod "d40a938f-1aa7-4897-b5a7-91257132baf7" (UID: "d40a938f-1aa7-4897-b5a7-91257132baf7"). InnerVolumeSpecName "kube-api-access-6hrg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:35.553040 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.552851 5801 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.553040 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.552887 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clk7h\" (UniqueName: \"kubernetes.io/projected/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-kube-api-access-clk7h\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.553040 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.552903 5801 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.553040 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.552919 5801 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.553040 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.552933 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hrg6\" (UniqueName: \"kubernetes.io/projected/d40a938f-1aa7-4897-b5a7-91257132baf7-kube-api-access-6hrg6\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.553040 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.552947 5801 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.553040 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.552960 5801 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d40a938f-1aa7-4897-b5a7-91257132baf7-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.553040 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.552973 5801 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.553040 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.552986 5801 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d40a938f-1aa7-4897-b5a7-91257132baf7-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.553040 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.553000 5801 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.553040 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.553015 5801 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.747116 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:35.747082 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:36.263861 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.263797 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" containerID="cri-o://c67da18cae2a5adcdb13dd347588da5a2def36174a2202e3158a4b77bbda2798" gracePeriod=45 Oct 30 21:51:36.308373 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.308336 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" containerID="cri-o://c8b66342f364a4cac46948e5cc97c677dc7bc29f72405ecc6c56bbc648a8911c" gracePeriod=45 Oct 30 21:51:36.312646 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.312621 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" Oct 30 21:51:36.312789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.312634 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842" event={"ID":"d40a938f-1aa7-4897-b5a7-91257132baf7","Type":"ContainerDied","Data":"b4b4d0ef1e25803e4c0b6f1fc7f76f0da2cdfa4818f537536d511fe5dc2f9109"} Oct 30 21:51:36.312789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.312688 5801 scope.go:122] "RemoveContainer" containerID="290f91d4dfd4a1e83f4363e8306afeb90b1a4dd2916a01c86bfe4abc5f6dd870" Oct 30 21:51:36.314272 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.314255 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" Oct 30 21:51:36.314358 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.314288 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr" event={"ID":"d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5","Type":"ContainerDied","Data":"8dbee447c3662f30fcd72dec7a75186ac15f7aca6df0961d51974a6a1a072ce7"} Oct 30 21:51:36.392411 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.392389 5801 scope.go:122] "RemoveContainer" containerID="18c8ea1eabde5238d47b8c4447efbafa9ab1177f7a7cb1b04acf4832e17a12ff" Oct 30 21:51:36.406857 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.406833 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842"] Oct 30 21:51:36.410056 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.410033 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-1a81ae/premium-w1-1a81ae-kserve-679cdd99b8-mf842"] Oct 30 21:51:36.412405 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.412372 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": dial tcp 10.133.0.74:8000: connect: connection refused" Oct 30 21:51:36.437771 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.437271 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr"] Oct 30 21:51:36.461413 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.461368 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-1a81ae/distinct-2-w1-1a81ae-kserve-7b6d4fd6b8-799hr"] Oct 30 21:51:36.500142 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.499914 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c67196-c2mnf" Oct 30 21:51:36.515946 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.515929 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c9eb0-9fb99" Oct 30 21:51:36.546399 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.546295 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:36.580147 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.580101 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a8a85ae-4b64-473a-beac-cbde16b4ea4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a8a85ae-4b64-473a-beac-cbde16b4ea4f-maas-api-xks-ca\") pod \"6a8a85ae-4b64-473a-beac-cbde16b4ea4f\" (UID: \"6a8a85ae-4b64-473a-beac-cbde16b4ea4f\") " Oct 30 21:51:36.580313 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.580265 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a8a85ae-4b64-473a-beac-cbde16b4ea4f-kube-api-access-6h5j4\" (UniqueName: \"kubernetes.io/projected/6a8a85ae-4b64-473a-beac-cbde16b4ea4f-kube-api-access-6h5j4\") pod \"6a8a85ae-4b64-473a-beac-cbde16b4ea4f\" (UID: \"6a8a85ae-4b64-473a-beac-cbde16b4ea4f\") " Oct 30 21:51:36.580387 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.580323 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a8a85ae-4b64-473a-beac-cbde16b4ea4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a8a85ae-4b64-473a-beac-cbde16b4ea4f-maas-api-openshift-ca\") pod \"6a8a85ae-4b64-473a-beac-cbde16b4ea4f\" (UID: \"6a8a85ae-4b64-473a-beac-cbde16b4ea4f\") " Oct 30 21:51:36.582375 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.582097 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a8a85ae-4b64-473a-beac-cbde16b4ea4f-maas-api-openshift-ca" pod "6a8a85ae-4b64-473a-beac-cbde16b4ea4f" (UID: "6a8a85ae-4b64-473a-beac-cbde16b4ea4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:36.609876 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.603122 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a8a85ae-4b64-473a-beac-cbde16b4ea4f-kube-api-access-6h5j4" pod "6a8a85ae-4b64-473a-beac-cbde16b4ea4f" (UID: "6a8a85ae-4b64-473a-beac-cbde16b4ea4f"). InnerVolumeSpecName "kube-api-access-6h5j4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:36.655898 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.655855 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a8a85ae-4b64-473a-beac-cbde16b4ea4f-maas-api-xks-ca" pod "6a8a85ae-4b64-473a-beac-cbde16b4ea4f" (UID: "6a8a85ae-4b64-473a-beac-cbde16b4ea4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:36.683170 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.683140 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9e4efc8-551c-4352-a036-030ba1117dc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9e4efc8-551c-4352-a036-030ba1117dc3-maas-api-xks-ca\") pod \"b9e4efc8-551c-4352-a036-030ba1117dc3\" (UID: \"b9e4efc8-551c-4352-a036-030ba1117dc3\") " Oct 30 21:51:36.683323 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.683212 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9e4efc8-551c-4352-a036-030ba1117dc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9e4efc8-551c-4352-a036-030ba1117dc3-maas-api-openshift-ca\") pod \"b9e4efc8-551c-4352-a036-030ba1117dc3\" (UID: \"b9e4efc8-551c-4352-a036-030ba1117dc3\") " Oct 30 21:51:36.683406 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.683358 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9e4efc8-551c-4352-a036-030ba1117dc3-kube-api-access-rb9l5\" (UniqueName: \"kubernetes.io/projected/b9e4efc8-551c-4352-a036-030ba1117dc3-kube-api-access-rb9l5\") pod \"b9e4efc8-551c-4352-a036-030ba1117dc3\" (UID: \"b9e4efc8-551c-4352-a036-030ba1117dc3\") " Oct 30 21:51:36.683843 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.683816 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e4efc8-551c-4352-a036-030ba1117dc3-maas-api-openshift-ca" pod "b9e4efc8-551c-4352-a036-030ba1117dc3" (UID: "b9e4efc8-551c-4352-a036-030ba1117dc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:36.684254 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.684224 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a8a85ae-4b64-473a-beac-cbde16b4ea4f-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:36.684373 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.684356 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a8a85ae-4b64-473a-beac-cbde16b4ea4f-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:36.684550 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.684377 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9e4efc8-551c-4352-a036-030ba1117dc3-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:36.684550 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.684394 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h5j4\" (UniqueName: \"kubernetes.io/projected/6a8a85ae-4b64-473a-beac-cbde16b4ea4f-kube-api-access-6h5j4\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:36.687794 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.687770 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e4efc8-551c-4352-a036-030ba1117dc3-kube-api-access-rb9l5" pod "b9e4efc8-551c-4352-a036-030ba1117dc3" (UID: "b9e4efc8-551c-4352-a036-030ba1117dc3"). InnerVolumeSpecName "kube-api-access-rb9l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:36.689021 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.688995 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9e4efc8-551c-4352-a036-030ba1117dc3-maas-api-xks-ca" pod "b9e4efc8-551c-4352-a036-030ba1117dc3" (UID: "b9e4efc8-551c-4352-a036-030ba1117dc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:36.712722 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.712697 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:51:36.716159 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.716136 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:51:36.785551 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.785469 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-home\" (UniqueName: \"kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-home\") pod \"c2ea2669-ebc8-484c-aaef-33298655e79e\" (UID: \"c2ea2669-ebc8-484c-aaef-33298655e79e\") " Oct 30 21:51:36.785551 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.785528 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-home\" (UniqueName: \"kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-home\") pod \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\" (UID: \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\") " Oct 30 21:51:36.785803 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.785596 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2ea2669-ebc8-484c-aaef-33298655e79e-tls-certs\" (UniqueName: \"kubernetes.io/secret/c2ea2669-ebc8-484c-aaef-33298655e79e-tls-certs\") pod \"c2ea2669-ebc8-484c-aaef-33298655e79e\" (UID: \"c2ea2669-ebc8-484c-aaef-33298655e79e\") " Oct 30 21:51:36.785803 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.785630 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-tmp-dir\") pod \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\" (UID: \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\") " Oct 30 21:51:36.785803 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.785667 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-dshm\") pod \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\" (UID: \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\") " Oct 30 21:51:36.785803 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.785727 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-model-cache\") pod \"c2ea2669-ebc8-484c-aaef-33298655e79e\" (UID: \"c2ea2669-ebc8-484c-aaef-33298655e79e\") " Oct 30 21:51:36.785803 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.785796 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2ea2669-ebc8-484c-aaef-33298655e79e-kube-api-access-xr8wq\" (UniqueName: \"kubernetes.io/projected/c2ea2669-ebc8-484c-aaef-33298655e79e-kube-api-access-xr8wq\") pod \"c2ea2669-ebc8-484c-aaef-33298655e79e\" (UID: \"c2ea2669-ebc8-484c-aaef-33298655e79e\") " Oct 30 21:51:36.786058 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.785841 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-model-cache\") pod \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\" (UID: \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\") " Oct 30 21:51:36.786058 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.785869 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab0b6e40-2877-48b5-9dba-d3e733b8471d-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0b6e40-2877-48b5-9dba-d3e733b8471d-tls-certs\") pod \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\" (UID: \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\") " Oct 30 21:51:36.786058 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.785929 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-dshm\") pod \"c2ea2669-ebc8-484c-aaef-33298655e79e\" (UID: \"c2ea2669-ebc8-484c-aaef-33298655e79e\") " Oct 30 21:51:36.786058 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.785964 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-tmp-dir\") pod \"c2ea2669-ebc8-484c-aaef-33298655e79e\" (UID: \"c2ea2669-ebc8-484c-aaef-33298655e79e\") " Oct 30 21:51:36.786058 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.786007 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0b6e40-2877-48b5-9dba-d3e733b8471d-kube-api-access-srtvj\" (UniqueName: \"kubernetes.io/projected/ab0b6e40-2877-48b5-9dba-d3e733b8471d-kube-api-access-srtvj\") pod \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\" (UID: \"ab0b6e40-2877-48b5-9dba-d3e733b8471d\") " Oct 30 21:51:36.786606 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.786581 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9e4efc8-551c-4352-a036-030ba1117dc3-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:36.786726 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.786610 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb9l5\" (UniqueName: \"kubernetes.io/projected/b9e4efc8-551c-4352-a036-030ba1117dc3-kube-api-access-rb9l5\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:36.787106 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.785792 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-home" pod "c2ea2669-ebc8-484c-aaef-33298655e79e" (UID: "c2ea2669-ebc8-484c-aaef-33298655e79e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:36.787222 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.786117 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-tmp-dir" pod "ab0b6e40-2877-48b5-9dba-d3e733b8471d" (UID: "ab0b6e40-2877-48b5-9dba-d3e733b8471d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:36.787222 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.786501 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-model-cache" pod "ab0b6e40-2877-48b5-9dba-d3e733b8471d" (UID: "ab0b6e40-2877-48b5-9dba-d3e733b8471d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:36.787222 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.786680 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-tmp-dir" pod "c2ea2669-ebc8-484c-aaef-33298655e79e" (UID: "c2ea2669-ebc8-484c-aaef-33298655e79e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:36.787222 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.787074 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-model-cache" pod "c2ea2669-ebc8-484c-aaef-33298655e79e" (UID: "c2ea2669-ebc8-484c-aaef-33298655e79e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:36.787454 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.787335 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-home" pod "ab0b6e40-2877-48b5-9dba-d3e733b8471d" (UID: "ab0b6e40-2877-48b5-9dba-d3e733b8471d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:36.788931 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.788901 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-dshm" pod "c2ea2669-ebc8-484c-aaef-33298655e79e" (UID: "c2ea2669-ebc8-484c-aaef-33298655e79e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:36.789609 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.789587 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0b6e40-2877-48b5-9dba-d3e733b8471d-kube-api-access-srtvj" pod "ab0b6e40-2877-48b5-9dba-d3e733b8471d" (UID: "ab0b6e40-2877-48b5-9dba-d3e733b8471d"). InnerVolumeSpecName "kube-api-access-srtvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:36.789736 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.789665 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ea2669-ebc8-484c-aaef-33298655e79e-kube-api-access-xr8wq" pod "c2ea2669-ebc8-484c-aaef-33298655e79e" (UID: "c2ea2669-ebc8-484c-aaef-33298655e79e"). InnerVolumeSpecName "kube-api-access-xr8wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:36.789982 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.789957 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-dshm" pod "ab0b6e40-2877-48b5-9dba-d3e733b8471d" (UID: "ab0b6e40-2877-48b5-9dba-d3e733b8471d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:36.790185 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.790110 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2ea2669-ebc8-484c-aaef-33298655e79e-tls-certs" pod "c2ea2669-ebc8-484c-aaef-33298655e79e" (UID: "c2ea2669-ebc8-484c-aaef-33298655e79e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:36.790410 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.790389 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab0b6e40-2877-48b5-9dba-d3e733b8471d-tls-certs" pod "ab0b6e40-2877-48b5-9dba-d3e733b8471d" (UID: "ab0b6e40-2877-48b5-9dba-d3e733b8471d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:36.887443 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.887414 5801 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2ea2669-ebc8-484c-aaef-33298655e79e-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:36.887443 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.887440 5801 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:36.887443 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.887449 5801 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:36.887643 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.887458 5801 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:36.887643 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.887470 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr8wq\" (UniqueName: \"kubernetes.io/projected/c2ea2669-ebc8-484c-aaef-33298655e79e-kube-api-access-xr8wq\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:36.887643 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.887478 5801 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:36.887643 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.887487 5801 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0b6e40-2877-48b5-9dba-d3e733b8471d-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:36.887643 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.887495 5801 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:36.887643 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.887504 5801 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:36.887643 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.887512 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srtvj\" (UniqueName: \"kubernetes.io/projected/ab0b6e40-2877-48b5-9dba-d3e733b8471d-kube-api-access-srtvj\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:36.887643 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.887520 5801 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2ea2669-ebc8-484c-aaef-33298655e79e-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:36.887643 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:36.887527 5801 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab0b6e40-2877-48b5-9dba-d3e733b8471d-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:37.325984 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.325941 5801 generic.go:356] "Generic (PLEG): container finished" podID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerID="c8b66342f364a4cac46948e5cc97c677dc7bc29f72405ecc6c56bbc648a8911c" exitCode=0 Oct 30 21:51:37.326974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.325990 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" Oct 30 21:51:37.326974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.326082 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" event={"ID":"c2ea2669-ebc8-484c-aaef-33298655e79e","Type":"ContainerDied","Data":"c8b66342f364a4cac46948e5cc97c677dc7bc29f72405ecc6c56bbc648a8911c"} Oct 30 21:51:37.326974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.326119 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb" event={"ID":"c2ea2669-ebc8-484c-aaef-33298655e79e","Type":"ContainerDied","Data":"8aed53f11cb7bcfcf75228fbe900d8cbf825764f63ee7e622af222411dd1a544"} Oct 30 21:51:37.326974 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.326484 5801 scope.go:122] "RemoveContainer" containerID="c8b66342f364a4cac46948e5cc97c677dc7bc29f72405ecc6c56bbc648a8911c" Oct 30 21:51:37.328236 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.328213 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c9eb0-9fb99" Oct 30 21:51:37.328236 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.328223 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c9eb0-9fb99" event={"ID":"b9e4efc8-551c-4352-a036-030ba1117dc3","Type":"ContainerDied","Data":"e75239999530ad56cd60d11acabd001b2067ba5ef3d84bda5bd4aca99c6ef2b1"} Oct 30 21:51:37.328366 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.328270 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e75239999530ad56cd60d11acabd001b2067ba5ef3d84bda5bd4aca99c6ef2b1" Oct 30 21:51:37.333215 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.333190 5801 generic.go:356] "Generic (PLEG): container finished" podID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerID="c67da18cae2a5adcdb13dd347588da5a2def36174a2202e3158a4b77bbda2798" exitCode=0 Oct 30 21:51:37.333332 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.333238 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" Oct 30 21:51:37.333332 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.333289 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" event={"ID":"ab0b6e40-2877-48b5-9dba-d3e733b8471d","Type":"ContainerDied","Data":"c67da18cae2a5adcdb13dd347588da5a2def36174a2202e3158a4b77bbda2798"} Oct 30 21:51:37.333447 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.333332 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f" event={"ID":"ab0b6e40-2877-48b5-9dba-d3e733b8471d","Type":"ContainerDied","Data":"744ff8c8b4f08f064b08e04c46d4227e7f6e1050f94b16ed3390ef564ab6e1b4"} Oct 30 21:51:37.338787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.335305 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c67196-c2mnf" event={"ID":"6a8a85ae-4b64-473a-beac-cbde16b4ea4f","Type":"ContainerDied","Data":"6fdda8b7d9c52a024fea49d1fb976f148b2ca89063a52fd215f1d5a0b3ba0674"} Oct 30 21:51:37.338787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.335335 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c67196-c2mnf" Oct 30 21:51:37.338787 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.335342 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fdda8b7d9c52a024fea49d1fb976f148b2ca89063a52fd215f1d5a0b3ba0674" Oct 30 21:51:37.343140 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.343123 5801 scope.go:122] "RemoveContainer" containerID="c8b66342f364a4cac46948e5cc97c677dc7bc29f72405ecc6c56bbc648a8911c" Oct 30 21:51:37.343407 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:37.343379 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b66342f364a4cac46948e5cc97c677dc7bc29f72405ecc6c56bbc648a8911c\": container with ID starting with c8b66342f364a4cac46948e5cc97c677dc7bc29f72405ecc6c56bbc648a8911c not found: ID does not exist" containerID="c8b66342f364a4cac46948e5cc97c677dc7bc29f72405ecc6c56bbc648a8911c" Oct 30 21:51:37.343508 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.343413 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b66342f364a4cac46948e5cc97c677dc7bc29f72405ecc6c56bbc648a8911c"} err="failed to get container status \"c8b66342f364a4cac46948e5cc97c677dc7bc29f72405ecc6c56bbc648a8911c\": rpc error: code = NotFound desc = could not find container \"c8b66342f364a4cac46948e5cc97c677dc7bc29f72405ecc6c56bbc648a8911c\": container with ID starting with c8b66342f364a4cac46948e5cc97c677dc7bc29f72405ecc6c56bbc648a8911c not found: ID does not exist" Oct 30 21:51:37.343508 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.343437 5801 scope.go:122] "RemoveContainer" containerID="c67da18cae2a5adcdb13dd347588da5a2def36174a2202e3158a4b77bbda2798" Oct 30 21:51:37.344016 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.343992 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:37.355980 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.355962 5801 scope.go:122] "RemoveContainer" containerID="c67da18cae2a5adcdb13dd347588da5a2def36174a2202e3158a4b77bbda2798" Oct 30 21:51:37.356260 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:37.356233 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c67da18cae2a5adcdb13dd347588da5a2def36174a2202e3158a4b77bbda2798\": container with ID starting with c67da18cae2a5adcdb13dd347588da5a2def36174a2202e3158a4b77bbda2798 not found: ID does not exist" containerID="c67da18cae2a5adcdb13dd347588da5a2def36174a2202e3158a4b77bbda2798" Oct 30 21:51:37.356324 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.356268 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c67da18cae2a5adcdb13dd347588da5a2def36174a2202e3158a4b77bbda2798"} err="failed to get container status \"c67da18cae2a5adcdb13dd347588da5a2def36174a2202e3158a4b77bbda2798\": rpc error: code = NotFound desc = could not find container \"c67da18cae2a5adcdb13dd347588da5a2def36174a2202e3158a4b77bbda2798\": container with ID starting with c67da18cae2a5adcdb13dd347588da5a2def36174a2202e3158a4b77bbda2798 not found: ID does not exist" Oct 30 21:51:37.358414 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.358392 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb"] Oct 30 21:51:37.364523 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.364502 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0fa864/simulator-w4-0fa864-kserve-657cf64664-d8hhb"] Oct 30 21:51:37.384278 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.384255 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f"] Oct 30 21:51:37.392712 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.392682 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0fa864/distinct-w4-0fa864-kserve-5977678676-pzd8f"] Oct 30 21:51:37.523179 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.523146 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" path="/var/lib/kubelet/pods/ab0b6e40-2877-48b5-9dba-d3e733b8471d/volumes" Oct 30 21:51:37.523645 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.523623 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" path="/var/lib/kubelet/pods/c2ea2669-ebc8-484c-aaef-33298655e79e/volumes" Oct 30 21:51:37.524098 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.524078 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" path="/var/lib/kubelet/pods/d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5/volumes" Oct 30 21:51:37.524819 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:37.524799 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" path="/var/lib/kubelet/pods/d40a938f-1aa7-4897-b5a7-91257132baf7/volumes" Oct 30 21:51:38.026362 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:38.026326 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:38.608217 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:38.608186 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:38.795510 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:38.795474 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:38.941780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:38.941096 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl"] Oct 30 21:51:38.957535 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:38.957035 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:38.968501 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:38.968468 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0fa864-x99cl"] Oct 30 21:51:39.185243 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:39.185202 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:39.372400 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:39.372365 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69"] Oct 30 21:51:39.372654 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:39.372624 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" podUID="457b1dc9-f94b-4dda-8b46-e125e0069033" containerName="istio-proxy" containerID="cri-o://1327543b60224dc1bd2af1573c0220b80df821174c3378522963597b4e7fa436" gracePeriod=30 Oct 30 21:51:39.418665 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:39.418633 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:39.535509 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:39.535458 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7de2823f-1b5e-434e-987a-133d6f101aa6" path="/var/lib/kubelet/pods/7de2823f-1b5e-434e-987a-133d6f101aa6/volumes" Oct 30 21:51:39.553139 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:39.552819 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:39.924602 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:39.924566 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:40.040880 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.040841 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:40.083962 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.083933 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6"] Oct 30 21:51:40.084379 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.084366 5801 cpu_manager.go:414] "containerMap: removing container" podUID="b9e4efc8-551c-4352-a036-030ba1117dc3" containerName="revoke-keys" Oct 30 21:51:40.084421 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.084380 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9e4efc8-551c-4352-a036-030ba1117dc3" containerName="revoke-keys" Oct 30 21:51:40.084421 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.084392 5801 cpu_manager.go:414] "containerMap: removing container" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" Oct 30 21:51:40.084421 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.084397 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" Oct 30 21:51:40.084421 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.084418 5801 cpu_manager.go:414] "containerMap: removing container" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" Oct 30 21:51:40.084545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.084424 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" Oct 30 21:51:40.084545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.084431 5801 cpu_manager.go:414] "containerMap: removing container" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" Oct 30 21:51:40.084545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.084435 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" Oct 30 21:51:40.084545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.084452 5801 cpu_manager.go:414] "containerMap: removing container" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" Oct 30 21:51:40.084545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.084456 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" Oct 30 21:51:40.084545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.084470 5801 cpu_manager.go:414] "containerMap: removing container" podUID="6a8a85ae-4b64-473a-beac-cbde16b4ea4f" containerName="revoke-keys" Oct 30 21:51:40.084545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.084475 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a8a85ae-4b64-473a-beac-cbde16b4ea4f" containerName="revoke-keys" Oct 30 21:51:40.084545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.084541 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="d40a938f-1aa7-4897-b5a7-91257132baf7" containerName="main" Oct 30 21:51:40.084545 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.084547 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2d6e8aa-9bf3-4078-99f8-e2f0be40d9b5" containerName="main" Oct 30 21:51:40.084851 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.084556 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0b6e40-2877-48b5-9dba-d3e733b8471d" containerName="main" Oct 30 21:51:40.084851 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.084565 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a8a85ae-4b64-473a-beac-cbde16b4ea4f" containerName="revoke-keys" Oct 30 21:51:40.084851 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.084572 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9e4efc8-551c-4352-a036-030ba1117dc3" containerName="revoke-keys" Oct 30 21:51:40.084851 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.084578 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2ea2669-ebc8-484c-aaef-33298655e79e" containerName="main" Oct 30 21:51:40.089392 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.089376 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6" Oct 30 21:51:40.096482 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.096460 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:51:40.107810 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.097896 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:51:40.115783 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.114997 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6"] Oct 30 21:51:40.137985 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.137954 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:51:40.137985 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.137955 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:51:40.155741 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.155710 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:40.234530 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.234446 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf8px\" (UniqueName: \"kubernetes.io/projected/fa0e622c-88d2-49b6-a7ea-8617987ee929-kube-api-access-wf8px\") pod \"maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6\" (UID: \"fa0e622c-88d2-49b6-a7ea-8617987ee929\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6" Oct 30 21:51:40.234668 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.234567 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0e622c-88d2-49b6-a7ea-8617987ee929-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6\" (UID: \"fa0e622c-88d2-49b6-a7ea-8617987ee929\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6" Oct 30 21:51:40.234668 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.234586 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0e622c-88d2-49b6-a7ea-8617987ee929-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6\" (UID: \"fa0e622c-88d2-49b6-a7ea-8617987ee929\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6" Oct 30 21:51:40.346741 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.346705 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf8px\" (UniqueName: \"kubernetes.io/projected/fa0e622c-88d2-49b6-a7ea-8617987ee929-kube-api-access-wf8px\") pod \"maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6\" (UID: \"fa0e622c-88d2-49b6-a7ea-8617987ee929\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6" Oct 30 21:51:40.347137 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.347109 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0e622c-88d2-49b6-a7ea-8617987ee929-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6\" (UID: \"fa0e622c-88d2-49b6-a7ea-8617987ee929\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6" Oct 30 21:51:40.347309 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.347292 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0e622c-88d2-49b6-a7ea-8617987ee929-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6\" (UID: \"fa0e622c-88d2-49b6-a7ea-8617987ee929\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6" Oct 30 21:51:40.348274 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.348251 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0e622c-88d2-49b6-a7ea-8617987ee929-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6\" (UID: \"fa0e622c-88d2-49b6-a7ea-8617987ee929\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6" Oct 30 21:51:40.353034 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.353011 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0e622c-88d2-49b6-a7ea-8617987ee929-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6\" (UID: \"fa0e622c-88d2-49b6-a7ea-8617987ee929\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6" Oct 30 21:51:40.382775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.382713 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z"] Oct 30 21:51:40.383034 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.382954 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" podUID="96f41b1e-ff7a-47b1-990a-677cdcd42a7f" containerName="istio-proxy" containerID="cri-o://6d48376b75ae2bf1826e0d48b34bc11740a011779f380450350401cc4d588e5c" gracePeriod=30 Oct 30 21:51:40.386922 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.386899 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf8px\" (UniqueName: \"kubernetes.io/projected/fa0e622c-88d2-49b6-a7ea-8617987ee929-kube-api-access-wf8px\") pod \"maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6\" (UID: \"fa0e622c-88d2-49b6-a7ea-8617987ee929\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6" Oct 30 21:51:40.410466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.410438 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6" Oct 30 21:51:40.410908 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.410873 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:40.706808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.706777 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6"] Oct 30 21:51:40.707344 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:51:40.707313 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0e622c_88d2_49b6_a7ea_8617987ee929.slice/crio-41f0192a50bfcfef702a59d1c8e463d6c4df6cb57c9fdf70287d614e8195470f WatchSource:0}: Error finding container 41f0192a50bfcfef702a59d1c8e463d6c4df6cb57c9fdf70287d614e8195470f: Status 404 returned error can't find the container with id 41f0192a50bfcfef702a59d1c8e463d6c4df6cb57c9fdf70287d614e8195470f Oct 30 21:51:40.950919 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:40.950884 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:41.110724 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.110684 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:41.208656 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.208619 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84"] Oct 30 21:51:41.212129 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.212103 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-1a81ae-sbb84"] Oct 30 21:51:41.221288 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.221253 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:41.349173 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.349130 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:41.390170 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.390074 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6" event={"ID":"fa0e622c-88d2-49b6-a7ea-8617987ee929","Type":"ContainerStarted","Data":"26e97e35207154a84f60d2c25740e764499296e737e774f1d0cbc4fae340f926"} Oct 30 21:51:41.390326 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.390169 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6" event={"ID":"fa0e622c-88d2-49b6-a7ea-8617987ee929","Type":"ContainerStarted","Data":"41f0192a50bfcfef702a59d1c8e463d6c4df6cb57c9fdf70287d614e8195470f"} Oct 30 21:51:41.413365 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.413299 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6" podStartSLOduration=1.4132806119999999 podStartE2EDuration="1.413280612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:41.410431437 +0000 UTC m=+1370.006248098" watchObservedRunningTime="2026-10-30 21:51:41.413280612 +0000 UTC m=+1370.009097273" Oct 30 21:51:41.469877 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.469823 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" containerID="cri-o://57ccf884c1ddf8e67ff6d99db1412fd71c1edce8a202df0175e22fcc2e964029" gracePeriod=45 Oct 30 21:51:41.470571 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.470171 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" containerID="cri-o://2e87a4e5ddf07a6d002b6aba59757416362357cbc088b588938d6511206d4c1d" gracePeriod=45 Oct 30 21:51:41.527947 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.527913 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c84c6d7e-7ce7-4caf-97ec-43267070cab3" path="/var/lib/kubelet/pods/c84c6d7e-7ce7-4caf-97ec-43267070cab3/volumes" Oct 30 21:51:41.720398 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.720377 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:51:41.743102 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.743069 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:41.762353 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.762326 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:51:41.871967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.871938 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-model-cache\") pod \"ea440bed-5030-4eeb-85b7-1916ffe802af\" (UID: \"ea440bed-5030-4eeb-85b7-1916ffe802af\") " Oct 30 21:51:41.872126 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.872003 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-model-cache\") pod \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\" (UID: \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\") " Oct 30 21:51:41.872126 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.872049 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-dshm\") pod \"ea440bed-5030-4eeb-85b7-1916ffe802af\" (UID: \"ea440bed-5030-4eeb-85b7-1916ffe802af\") " Oct 30 21:51:41.872126 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.872090 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-tmp-dir\") pod \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\" (UID: \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\") " Oct 30 21:51:41.872126 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.872120 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-kube-api-access-lrldb\" (UniqueName: \"kubernetes.io/projected/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-kube-api-access-lrldb\") pod \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\" (UID: \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\") " Oct 30 21:51:41.872364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.872140 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-home\" (UniqueName: \"kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-home\") pod \"ea440bed-5030-4eeb-85b7-1916ffe802af\" (UID: \"ea440bed-5030-4eeb-85b7-1916ffe802af\") " Oct 30 21:51:41.872364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.872169 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-tls-certs\" (UniqueName: \"kubernetes.io/secret/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-tls-certs\") pod \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\" (UID: \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\") " Oct 30 21:51:41.872364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.872242 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-model-cache" pod "ea440bed-5030-4eeb-85b7-1916ffe802af" (UID: "ea440bed-5030-4eeb-85b7-1916ffe802af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:41.872364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.872279 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-model-cache" pod "363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" (UID: "363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:41.872364 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.872329 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea440bed-5030-4eeb-85b7-1916ffe802af-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea440bed-5030-4eeb-85b7-1916ffe802af-tls-certs\") pod \"ea440bed-5030-4eeb-85b7-1916ffe802af\" (UID: \"ea440bed-5030-4eeb-85b7-1916ffe802af\") " Oct 30 21:51:41.872743 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.872405 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-tmp-dir" pod "363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" (UID: "363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:41.872743 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.872450 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-home" pod "ea440bed-5030-4eeb-85b7-1916ffe802af" (UID: "ea440bed-5030-4eeb-85b7-1916ffe802af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:41.872743 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.872598 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-home" pod "363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" (UID: "363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:41.872743 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.872633 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-home\" (UniqueName: \"kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-home\") pod \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\" (UID: \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\") " Oct 30 21:51:41.872743 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.872702 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-dshm\") pod \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\" (UID: \"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb\") " Oct 30 21:51:41.873054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.872746 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-tmp-dir\") pod \"ea440bed-5030-4eeb-85b7-1916ffe802af\" (UID: \"ea440bed-5030-4eeb-85b7-1916ffe802af\") " Oct 30 21:51:41.873054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.872812 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea440bed-5030-4eeb-85b7-1916ffe802af-kube-api-access-2qr2j\" (UniqueName: \"kubernetes.io/projected/ea440bed-5030-4eeb-85b7-1916ffe802af-kube-api-access-2qr2j\") pod \"ea440bed-5030-4eeb-85b7-1916ffe802af\" (UID: \"ea440bed-5030-4eeb-85b7-1916ffe802af\") " Oct 30 21:51:41.873168 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.873047 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-tmp-dir" pod "ea440bed-5030-4eeb-85b7-1916ffe802af" (UID: "ea440bed-5030-4eeb-85b7-1916ffe802af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:41.873470 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.873441 5801 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:41.873470 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.873472 5801 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:41.873658 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.873488 5801 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:41.873658 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.873503 5801 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:41.873658 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.873516 5801 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:41.873658 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.873567 5801 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:41.874675 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.874651 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea440bed-5030-4eeb-85b7-1916ffe802af-tls-certs" pod "ea440bed-5030-4eeb-85b7-1916ffe802af" (UID: "ea440bed-5030-4eeb-85b7-1916ffe802af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:41.874855 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.874833 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-tls-certs" pod "363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" (UID: "363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:41.874930 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.874863 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-dshm" pod "ea440bed-5030-4eeb-85b7-1916ffe802af" (UID: "ea440bed-5030-4eeb-85b7-1916ffe802af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:41.874930 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.874914 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-kube-api-access-lrldb" pod "363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" (UID: "363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb"). InnerVolumeSpecName "kube-api-access-lrldb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:41.875044 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.874983 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-dshm" pod "363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" (UID: "363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:41.875371 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.875343 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea440bed-5030-4eeb-85b7-1916ffe802af-kube-api-access-2qr2j" pod "ea440bed-5030-4eeb-85b7-1916ffe802af" (UID: "ea440bed-5030-4eeb-85b7-1916ffe802af"). InnerVolumeSpecName "kube-api-access-2qr2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:41.975088 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.975017 5801 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea440bed-5030-4eeb-85b7-1916ffe802af-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:41.975088 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.975042 5801 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:41.975088 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.975052 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qr2j\" (UniqueName: \"kubernetes.io/projected/ea440bed-5030-4eeb-85b7-1916ffe802af-kube-api-access-2qr2j\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:41.975088 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.975061 5801 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea440bed-5030-4eeb-85b7-1916ffe802af-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:41.975088 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.975070 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrldb\" (UniqueName: \"kubernetes.io/projected/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-kube-api-access-lrldb\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:41.975088 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:41.975079 5801 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:42.133022 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:42.132983 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:42.146992 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:42.146962 5801 patch_prober.go:29] interesting pod/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:51:42.147145 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:42.147009 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" podUID="96f41b1e-ff7a-47b1-990a-677cdcd42a7f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:51:42.407815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:42.407745 5801 generic.go:356] "Generic (PLEG): container finished" podID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerID="2e87a4e5ddf07a6d002b6aba59757416362357cbc088b588938d6511206d4c1d" exitCode=0 Oct 30 21:51:42.407973 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:42.407820 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" Oct 30 21:51:42.407973 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:42.407854 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" event={"ID":"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb","Type":"ContainerDied","Data":"2e87a4e5ddf07a6d002b6aba59757416362357cbc088b588938d6511206d4c1d"} Oct 30 21:51:42.407973 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:42.407897 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5" event={"ID":"363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb","Type":"ContainerDied","Data":"2cda24714eefab85122abf3c79f9c6e49b7692bba2378421b399bf1a1057cd51"} Oct 30 21:51:42.407973 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:42.407914 5801 scope.go:122] "RemoveContainer" containerID="2e87a4e5ddf07a6d002b6aba59757416362357cbc088b588938d6511206d4c1d" Oct 30 21:51:42.409650 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:42.409622 5801 generic.go:356] "Generic (PLEG): container finished" podID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerID="57ccf884c1ddf8e67ff6d99db1412fd71c1edce8a202df0175e22fcc2e964029" exitCode=0 Oct 30 21:51:42.409851 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:42.409684 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" Oct 30 21:51:42.409851 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:42.409715 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" event={"ID":"ea440bed-5030-4eeb-85b7-1916ffe802af","Type":"ContainerDied","Data":"57ccf884c1ddf8e67ff6d99db1412fd71c1edce8a202df0175e22fcc2e964029"} Oct 30 21:51:42.409851 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:42.409766 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq" event={"ID":"ea440bed-5030-4eeb-85b7-1916ffe802af","Type":"ContainerDied","Data":"55463fafe334facd8cde88f781fe86c84bab9e31e3389ab156b2fefe0e972d80"} Oct 30 21:51:42.412720 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:42.411737 5801 generic.go:356] "Generic (PLEG): container finished" podID="fa0e622c-88d2-49b6-a7ea-8617987ee929" containerID="26e97e35207154a84f60d2c25740e764499296e737e774f1d0cbc4fae340f926" exitCode=0 Oct 30 21:51:42.412720 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:42.411862 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6" event={"ID":"fa0e622c-88d2-49b6-a7ea-8617987ee929","Type":"ContainerDied","Data":"26e97e35207154a84f60d2c25740e764499296e737e774f1d0cbc4fae340f926"} Oct 30 21:51:42.419028 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:42.419007 5801 scope.go:122] "RemoveContainer" containerID="2e87a4e5ddf07a6d002b6aba59757416362357cbc088b588938d6511206d4c1d" Oct 30 21:51:42.419294 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:42.419268 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e87a4e5ddf07a6d002b6aba59757416362357cbc088b588938d6511206d4c1d\": container with ID starting with 2e87a4e5ddf07a6d002b6aba59757416362357cbc088b588938d6511206d4c1d not found: ID does not exist" containerID="2e87a4e5ddf07a6d002b6aba59757416362357cbc088b588938d6511206d4c1d" Oct 30 21:51:42.419337 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:42.419300 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e87a4e5ddf07a6d002b6aba59757416362357cbc088b588938d6511206d4c1d"} err="failed to get container status \"2e87a4e5ddf07a6d002b6aba59757416362357cbc088b588938d6511206d4c1d\": rpc error: code = NotFound desc = could not find container \"2e87a4e5ddf07a6d002b6aba59757416362357cbc088b588938d6511206d4c1d\": container with ID starting with 2e87a4e5ddf07a6d002b6aba59757416362357cbc088b588938d6511206d4c1d not found: ID does not exist" Oct 30 21:51:42.419337 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:42.419323 5801 scope.go:122] "RemoveContainer" containerID="57ccf884c1ddf8e67ff6d99db1412fd71c1edce8a202df0175e22fcc2e964029" Oct 30 21:51:42.428536 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:42.428512 5801 scope.go:122] "RemoveContainer" containerID="57ccf884c1ddf8e67ff6d99db1412fd71c1edce8a202df0175e22fcc2e964029" Oct 30 21:51:42.428817 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:42.428783 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57ccf884c1ddf8e67ff6d99db1412fd71c1edce8a202df0175e22fcc2e964029\": container with ID starting with 57ccf884c1ddf8e67ff6d99db1412fd71c1edce8a202df0175e22fcc2e964029 not found: ID does not exist" containerID="57ccf884c1ddf8e67ff6d99db1412fd71c1edce8a202df0175e22fcc2e964029" Oct 30 21:51:42.428817 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:42.428809 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ccf884c1ddf8e67ff6d99db1412fd71c1edce8a202df0175e22fcc2e964029"} err="failed to get container status \"57ccf884c1ddf8e67ff6d99db1412fd71c1edce8a202df0175e22fcc2e964029\": rpc error: code = NotFound desc = could not find container \"57ccf884c1ddf8e67ff6d99db1412fd71c1edce8a202df0175e22fcc2e964029\": container with ID starting with 57ccf884c1ddf8e67ff6d99db1412fd71c1edce8a202df0175e22fcc2e964029 not found: ID does not exist" Oct 30 21:51:42.451710 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:42.451680 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5"] Oct 30 21:51:42.455174 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:42.455151 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-1d8034/simulator-w3-1d8034-kserve-8648d847b4-g2qr5"] Oct 30 21:51:42.474175 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:42.474152 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq"] Oct 30 21:51:42.478715 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:42.478694 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-1d8034/premium-w3-1d8034-kserve-76c7f96f45-hsqbq"] Oct 30 21:51:43.167781 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:43.167733 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:43.526376 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:43.526331 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" path="/var/lib/kubelet/pods/363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb/volumes" Oct 30 21:51:43.527050 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:43.527024 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" path="/var/lib/kubelet/pods/ea440bed-5030-4eeb-85b7-1916ffe802af/volumes" Oct 30 21:51:43.573185 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:43.573162 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6" Oct 30 21:51:43.695613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:43.695544 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa0e622c-88d2-49b6-a7ea-8617987ee929-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0e622c-88d2-49b6-a7ea-8617987ee929-maas-api-xks-ca\") pod \"fa0e622c-88d2-49b6-a7ea-8617987ee929\" (UID: \"fa0e622c-88d2-49b6-a7ea-8617987ee929\") " Oct 30 21:51:43.695613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:43.695598 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa0e622c-88d2-49b6-a7ea-8617987ee929-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0e622c-88d2-49b6-a7ea-8617987ee929-maas-api-openshift-ca\") pod \"fa0e622c-88d2-49b6-a7ea-8617987ee929\" (UID: \"fa0e622c-88d2-49b6-a7ea-8617987ee929\") " Oct 30 21:51:43.695817 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:43.695789 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa0e622c-88d2-49b6-a7ea-8617987ee929-kube-api-access-wf8px\" (UniqueName: \"kubernetes.io/projected/fa0e622c-88d2-49b6-a7ea-8617987ee929-kube-api-access-wf8px\") pod \"fa0e622c-88d2-49b6-a7ea-8617987ee929\" (UID: \"fa0e622c-88d2-49b6-a7ea-8617987ee929\") " Oct 30 21:51:43.695998 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:43.695965 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0e622c-88d2-49b6-a7ea-8617987ee929-maas-api-openshift-ca" pod "fa0e622c-88d2-49b6-a7ea-8617987ee929" (UID: "fa0e622c-88d2-49b6-a7ea-8617987ee929"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:43.696405 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:43.696382 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0e622c-88d2-49b6-a7ea-8617987ee929-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:43.697739 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:43.697715 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa0e622c-88d2-49b6-a7ea-8617987ee929-maas-api-xks-ca" pod "fa0e622c-88d2-49b6-a7ea-8617987ee929" (UID: "fa0e622c-88d2-49b6-a7ea-8617987ee929"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:43.697851 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:43.697778 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0e622c-88d2-49b6-a7ea-8617987ee929-kube-api-access-wf8px" pod "fa0e622c-88d2-49b6-a7ea-8617987ee929" (UID: "fa0e622c-88d2-49b6-a7ea-8617987ee929"). InnerVolumeSpecName "kube-api-access-wf8px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:43.797949 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:43.797924 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0e622c-88d2-49b6-a7ea-8617987ee929-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:43.797949 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:43.797949 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf8px\" (UniqueName: \"kubernetes.io/projected/fa0e622c-88d2-49b6-a7ea-8617987ee929-kube-api-access-wf8px\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:43.937387 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:43.937354 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:43.971534 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:43.971458 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw" podUID="abc6bc4f-b6fe-49aa-9f79-41da164ffa08" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:51:44.432796 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.432741 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6" Oct 30 21:51:44.432796 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.432779 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6" event={"ID":"fa0e622c-88d2-49b6-a7ea-8617987ee929","Type":"ContainerDied","Data":"41f0192a50bfcfef702a59d1c8e463d6c4df6cb57c9fdf70287d614e8195470f"} Oct 30 21:51:44.433161 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.432826 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41f0192a50bfcfef702a59d1c8e463d6c4df6cb57c9fdf70287d614e8195470f" Oct 30 21:51:44.620635 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.620611 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:51:44.708183 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.708114 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-workload-socket\") pod \"457b1dc9-f94b-4dda-8b46-e125e0069033\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " Oct 30 21:51:44.708183 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.708175 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-token\" (UniqueName: \"kubernetes.io/projected/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-token\") pod \"457b1dc9-f94b-4dda-8b46-e125e0069033\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " Oct 30 21:51:44.708367 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.708202 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-data\") pod \"457b1dc9-f94b-4dda-8b46-e125e0069033\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " Oct 30 21:51:44.708367 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.708224 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-envoy\") pod \"457b1dc9-f94b-4dda-8b46-e125e0069033\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " Oct 30 21:51:44.708367 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.708249 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/457b1dc9-f94b-4dda-8b46-e125e0069033-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/457b1dc9-f94b-4dda-8b46-e125e0069033-istiod-ca-cert\") pod \"457b1dc9-f94b-4dda-8b46-e125e0069033\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " Oct 30 21:51:44.708367 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.708318 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-podinfo\") pod \"457b1dc9-f94b-4dda-8b46-e125e0069033\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " Oct 30 21:51:44.708367 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.708353 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-credential-socket\") pod \"457b1dc9-f94b-4dda-8b46-e125e0069033\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " Oct 30 21:51:44.708694 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.708393 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/457b1dc9-f94b-4dda-8b46-e125e0069033-kube-api-access-rc8sh\" (UniqueName: \"kubernetes.io/projected/457b1dc9-f94b-4dda-8b46-e125e0069033-kube-api-access-rc8sh\") pod \"457b1dc9-f94b-4dda-8b46-e125e0069033\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " Oct 30 21:51:44.708694 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.708407 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-workload-socket" pod "457b1dc9-f94b-4dda-8b46-e125e0069033" (UID: "457b1dc9-f94b-4dda-8b46-e125e0069033"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:44.708694 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.708434 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-workload-certs\") pod \"457b1dc9-f94b-4dda-8b46-e125e0069033\" (UID: \"457b1dc9-f94b-4dda-8b46-e125e0069033\") " Oct 30 21:51:44.708694 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.708534 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-data" pod "457b1dc9-f94b-4dda-8b46-e125e0069033" (UID: "457b1dc9-f94b-4dda-8b46-e125e0069033"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:44.708961 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.708700 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-credential-socket" pod "457b1dc9-f94b-4dda-8b46-e125e0069033" (UID: "457b1dc9-f94b-4dda-8b46-e125e0069033"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:44.708961 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.708820 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/457b1dc9-f94b-4dda-8b46-e125e0069033-istiod-ca-cert" pod "457b1dc9-f94b-4dda-8b46-e125e0069033" (UID: "457b1dc9-f94b-4dda-8b46-e125e0069033"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:44.709062 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.708990 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-workload-certs" pod "457b1dc9-f94b-4dda-8b46-e125e0069033" (UID: "457b1dc9-f94b-4dda-8b46-e125e0069033"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:44.709126 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.709059 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:44.709126 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.709084 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:44.709126 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.709099 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/457b1dc9-f94b-4dda-8b46-e125e0069033-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:44.709126 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.709113 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:44.710528 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.710500 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-envoy" pod "457b1dc9-f94b-4dda-8b46-e125e0069033" (UID: "457b1dc9-f94b-4dda-8b46-e125e0069033"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:44.710640 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.710563 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-token" pod "457b1dc9-f94b-4dda-8b46-e125e0069033" (UID: "457b1dc9-f94b-4dda-8b46-e125e0069033"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:44.711054 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.711030 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-podinfo" pod "457b1dc9-f94b-4dda-8b46-e125e0069033" (UID: "457b1dc9-f94b-4dda-8b46-e125e0069033"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:44.711150 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.711054 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457b1dc9-f94b-4dda-8b46-e125e0069033-kube-api-access-rc8sh" pod "457b1dc9-f94b-4dda-8b46-e125e0069033" (UID: "457b1dc9-f94b-4dda-8b46-e125e0069033"). InnerVolumeSpecName "kube-api-access-rc8sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:44.736374 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.736329 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:44.810000 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.809979 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:44.810083 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.810003 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:44.810083 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.810015 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/457b1dc9-f94b-4dda-8b46-e125e0069033-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:44.810083 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.810025 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc8sh\" (UniqueName: \"kubernetes.io/projected/457b1dc9-f94b-4dda-8b46-e125e0069033-kube-api-access-rc8sh\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:44.810083 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.810035 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/457b1dc9-f94b-4dda-8b46-e125e0069033-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:44.890730 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.890693 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p"] Oct 30 21:51:44.906385 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.906335 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb"] Oct 30 21:51:44.978153 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:44.978053 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:45.442694 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.442664 5801 generic.go:356] "Generic (PLEG): container finished" podID="457b1dc9-f94b-4dda-8b46-e125e0069033" containerID="1327543b60224dc1bd2af1573c0220b80df821174c3378522963597b4e7fa436" exitCode=0 Oct 30 21:51:45.443117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.442741 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" Oct 30 21:51:45.443117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.442767 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" event={"ID":"457b1dc9-f94b-4dda-8b46-e125e0069033","Type":"ContainerDied","Data":"1327543b60224dc1bd2af1573c0220b80df821174c3378522963597b4e7fa436"} Oct 30 21:51:45.443117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.442825 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69" event={"ID":"457b1dc9-f94b-4dda-8b46-e125e0069033","Type":"ContainerDied","Data":"10d7bbd534188a89c44218e72a866bc0726c385679b92e04d99a2668856deb12"} Oct 30 21:51:45.443117 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.442856 5801 scope.go:122] "RemoveContainer" containerID="1327543b60224dc1bd2af1573c0220b80df821174c3378522963597b4e7fa436" Oct 30 21:51:45.453233 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.453210 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:45.480665 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.480642 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69"] Oct 30 21:51:45.485066 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.485039 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0c62ab-openshift-default-76b4b6d87-79g69"] Oct 30 21:51:45.505043 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.504844 5801 scope.go:122] "RemoveContainer" containerID="0046d3214c1887638295980b75d32b3ee903884b4ca8bbcb13b97e894e8aca52" Oct 30 21:51:45.521410 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.520361 5801 scope.go:122] "RemoveContainer" containerID="1327543b60224dc1bd2af1573c0220b80df821174c3378522963597b4e7fa436" Oct 30 21:51:45.524446 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:45.522133 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1327543b60224dc1bd2af1573c0220b80df821174c3378522963597b4e7fa436\": container with ID starting with 1327543b60224dc1bd2af1573c0220b80df821174c3378522963597b4e7fa436 not found: ID does not exist" containerID="1327543b60224dc1bd2af1573c0220b80df821174c3378522963597b4e7fa436" Oct 30 21:51:45.524446 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.522179 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1327543b60224dc1bd2af1573c0220b80df821174c3378522963597b4e7fa436"} err="failed to get container status \"1327543b60224dc1bd2af1573c0220b80df821174c3378522963597b4e7fa436\": rpc error: code = NotFound desc = could not find container \"1327543b60224dc1bd2af1573c0220b80df821174c3378522963597b4e7fa436\": container with ID starting with 1327543b60224dc1bd2af1573c0220b80df821174c3378522963597b4e7fa436 not found: ID does not exist" Oct 30 21:51:45.524446 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.522231 5801 scope.go:122] "RemoveContainer" containerID="0046d3214c1887638295980b75d32b3ee903884b4ca8bbcb13b97e894e8aca52" Oct 30 21:51:45.524446 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:45.523143 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0046d3214c1887638295980b75d32b3ee903884b4ca8bbcb13b97e894e8aca52\": container with ID starting with 0046d3214c1887638295980b75d32b3ee903884b4ca8bbcb13b97e894e8aca52 not found: ID does not exist" containerID="0046d3214c1887638295980b75d32b3ee903884b4ca8bbcb13b97e894e8aca52" Oct 30 21:51:45.524446 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.523175 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0046d3214c1887638295980b75d32b3ee903884b4ca8bbcb13b97e894e8aca52"} err="failed to get container status \"0046d3214c1887638295980b75d32b3ee903884b4ca8bbcb13b97e894e8aca52\": rpc error: code = NotFound desc = could not find container \"0046d3214c1887638295980b75d32b3ee903884b4ca8bbcb13b97e894e8aca52\": container with ID starting with 0046d3214c1887638295980b75d32b3ee903884b4ca8bbcb13b97e894e8aca52 not found: ID does not exist" Oct 30 21:51:45.547625 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.547585 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="457b1dc9-f94b-4dda-8b46-e125e0069033" path="/var/lib/kubelet/pods/457b1dc9-f94b-4dda-8b46-e125e0069033/volumes" Oct 30 21:51:45.665677 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.665620 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:51:45.723792 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.723719 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-podinfo\") pod \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " Oct 30 21:51:45.724043 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.723835 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istiod-ca-cert\") pod \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " Oct 30 21:51:45.724258 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.724229 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-token\" (UniqueName: \"kubernetes.io/projected/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-token\") pod \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " Oct 30 21:51:45.724482 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.724284 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-credential-socket\") pod \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " Oct 30 21:51:45.724482 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.724351 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-envoy\") pod \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " Oct 30 21:51:45.724482 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.724378 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-workload-certs\") pod \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " Oct 30 21:51:45.724482 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.724404 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-data\") pod \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " Oct 30 21:51:45.724876 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.724551 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-kube-api-access-rv98j\" (UniqueName: \"kubernetes.io/projected/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-kube-api-access-rv98j\") pod \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " Oct 30 21:51:45.724876 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.724611 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-workload-socket\") pod \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\" (UID: \"96f41b1e-ff7a-47b1-990a-677cdcd42a7f\") " Oct 30 21:51:45.725300 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.724929 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istiod-ca-cert" pod "96f41b1e-ff7a-47b1-990a-677cdcd42a7f" (UID: "96f41b1e-ff7a-47b1-990a-677cdcd42a7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:45.725300 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.725194 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:45.725439 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.725394 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-workload-certs" pod "96f41b1e-ff7a-47b1-990a-677cdcd42a7f" (UID: "96f41b1e-ff7a-47b1-990a-677cdcd42a7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:45.725590 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.725570 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-data" pod "96f41b1e-ff7a-47b1-990a-677cdcd42a7f" (UID: "96f41b1e-ff7a-47b1-990a-677cdcd42a7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:45.726183 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.726105 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-credential-socket" pod "96f41b1e-ff7a-47b1-990a-677cdcd42a7f" (UID: "96f41b1e-ff7a-47b1-990a-677cdcd42a7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:45.727083 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.726939 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-workload-socket" pod "96f41b1e-ff7a-47b1-990a-677cdcd42a7f" (UID: "96f41b1e-ff7a-47b1-990a-677cdcd42a7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:45.730021 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.729890 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-envoy" pod "96f41b1e-ff7a-47b1-990a-677cdcd42a7f" (UID: "96f41b1e-ff7a-47b1-990a-677cdcd42a7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:45.730133 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.730045 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-podinfo" pod "96f41b1e-ff7a-47b1-990a-677cdcd42a7f" (UID: "96f41b1e-ff7a-47b1-990a-677cdcd42a7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:45.730223 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.730190 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-kube-api-access-rv98j" pod "96f41b1e-ff7a-47b1-990a-677cdcd42a7f" (UID: "96f41b1e-ff7a-47b1-990a-677cdcd42a7f"). InnerVolumeSpecName "kube-api-access-rv98j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:45.731314 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.731293 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-token" pod "96f41b1e-ff7a-47b1-990a-677cdcd42a7f" (UID: "96f41b1e-ff7a-47b1-990a-677cdcd42a7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:45.826156 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.826122 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:45.826156 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.826150 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:45.826156 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.826160 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:45.826427 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.826170 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv98j\" (UniqueName: \"kubernetes.io/projected/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-kube-api-access-rv98j\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:45.826427 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.826180 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:45.826427 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.826188 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:45.826427 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.826197 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:45.826427 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:45.826206 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f41b1e-ff7a-47b1-990a-677cdcd42a7f-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:46.158396 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:46.158354 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:46.453830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:46.453721 5801 generic.go:356] "Generic (PLEG): container finished" podID="96f41b1e-ff7a-47b1-990a-677cdcd42a7f" containerID="6d48376b75ae2bf1826e0d48b34bc11740a011779f380450350401cc4d588e5c" exitCode=0 Oct 30 21:51:46.453830 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:46.453808 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" Oct 30 21:51:46.454316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:46.453799 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" event={"ID":"96f41b1e-ff7a-47b1-990a-677cdcd42a7f","Type":"ContainerDied","Data":"6d48376b75ae2bf1826e0d48b34bc11740a011779f380450350401cc4d588e5c"} Oct 30 21:51:46.454316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:46.453947 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z" event={"ID":"96f41b1e-ff7a-47b1-990a-677cdcd42a7f","Type":"ContainerDied","Data":"8c7aaacb0e7317fcbd4d7868c1017f966eb5e4ceeece23434d04a608d6657c21"} Oct 30 21:51:46.454316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:46.453986 5801 scope.go:122] "RemoveContainer" containerID="6d48376b75ae2bf1826e0d48b34bc11740a011779f380450350401cc4d588e5c" Oct 30 21:51:46.465951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:46.465933 5801 scope.go:122] "RemoveContainer" containerID="6d48376b75ae2bf1826e0d48b34bc11740a011779f380450350401cc4d588e5c" Oct 30 21:51:46.466222 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:46.466198 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d48376b75ae2bf1826e0d48b34bc11740a011779f380450350401cc4d588e5c\": container with ID starting with 6d48376b75ae2bf1826e0d48b34bc11740a011779f380450350401cc4d588e5c not found: ID does not exist" containerID="6d48376b75ae2bf1826e0d48b34bc11740a011779f380450350401cc4d588e5c" Oct 30 21:51:46.466314 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:46.466228 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d48376b75ae2bf1826e0d48b34bc11740a011779f380450350401cc4d588e5c"} err="failed to get container status \"6d48376b75ae2bf1826e0d48b34bc11740a011779f380450350401cc4d588e5c\": rpc error: code = NotFound desc = could not find container \"6d48376b75ae2bf1826e0d48b34bc11740a011779f380450350401cc4d588e5c\": container with ID starting with 6d48376b75ae2bf1826e0d48b34bc11740a011779f380450350401cc4d588e5c not found: ID does not exist" Oct 30 21:51:46.481172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:46.481148 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z"] Oct 30 21:51:46.483994 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:46.483973 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-db0b54-openshift-default-79c79b9575-kqx2z"] Oct 30 21:51:46.959951 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:46.959918 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:47.076491 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:47.076448 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:47.189417 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:47.189378 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:47.523849 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:47.523816 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96f41b1e-ff7a-47b1-990a-677cdcd42a7f" path="/var/lib/kubelet/pods/96f41b1e-ff7a-47b1-990a-677cdcd42a7f/volumes" Oct 30 21:51:47.547335 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:47.547287 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:47.658167 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:47.658135 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:47.839298 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:47.839262 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:47.949416 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:47.949381 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:48.045886 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:48.045850 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:48.237814 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:48.235214 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd"] Oct 30 21:51:48.237814 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:48.235487 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" podUID="9a837073-7227-4ae9-9338-a70202213538" containerName="istio-proxy" containerID="cri-o://bd461a09ddfde0ed4c2332971e30b94c12f1ecb322523dd7ad8d581e0ad2cbc3" gracePeriod=30 Oct 30 21:51:48.244288 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:48.244251 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl"] Oct 30 21:51:48.244527 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:48.244499 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" podUID="0f258bc9-d2fd-446a-9154-36caf951e081" containerName="istio-proxy" containerID="cri-o://91184ee5cdc7e7d95c441873ef8ab2d8bbe6cf53bd147cd47d8682d422463155" gracePeriod=30 Oct 30 21:51:48.352848 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:48.352645 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:49.137574 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:49.137532 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:49.626268 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:49.626228 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:49.977353 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:49.977270 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:50.307693 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:50.307652 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:50.397472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:50.397435 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5"] Oct 30 21:51:50.400540 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:50.400510 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d8034-m67c5"] Oct 30 21:51:50.415834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:50.415803 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:50.494833 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:50.494800 5801 generic.go:356] "Generic (PLEG): container finished" podID="abc6bc4f-b6fe-49aa-9f79-41da164ffa08" containerID="14157862f3c04e54eb7387cdfb2527aaf664321bcb8403c105ce4962fd1d228d" exitCode=0 Oct 30 21:51:50.495085 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:50.494860 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw" event={"ID":"abc6bc4f-b6fe-49aa-9f79-41da164ffa08","Type":"ContainerDied","Data":"14157862f3c04e54eb7387cdfb2527aaf664321bcb8403c105ce4962fd1d228d"} Oct 30 21:51:50.558165 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:50.558143 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw" Oct 30 21:51:50.582249 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:50.582217 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc6bc4f-b6fe-49aa-9f79-41da164ffa08-kube-api-access-56xsr\" (UniqueName: \"kubernetes.io/projected/abc6bc4f-b6fe-49aa-9f79-41da164ffa08-kube-api-access-56xsr\") pod \"abc6bc4f-b6fe-49aa-9f79-41da164ffa08\" (UID: \"abc6bc4f-b6fe-49aa-9f79-41da164ffa08\") " Oct 30 21:51:50.582398 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:50.582275 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc6bc4f-b6fe-49aa-9f79-41da164ffa08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc6bc4f-b6fe-49aa-9f79-41da164ffa08-plugins-config-volume\") pod \"abc6bc4f-b6fe-49aa-9f79-41da164ffa08\" (UID: \"abc6bc4f-b6fe-49aa-9f79-41da164ffa08\") " Oct 30 21:51:50.582682 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:50.582656 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc6bc4f-b6fe-49aa-9f79-41da164ffa08-plugins-config-volume" pod "abc6bc4f-b6fe-49aa-9f79-41da164ffa08" (UID: "abc6bc4f-b6fe-49aa-9f79-41da164ffa08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:50.583171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:50.583145 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:50.584521 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:50.584500 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc6bc4f-b6fe-49aa-9f79-41da164ffa08-kube-api-access-56xsr" pod "abc6bc4f-b6fe-49aa-9f79-41da164ffa08" (UID: "abc6bc4f-b6fe-49aa-9f79-41da164ffa08"). InnerVolumeSpecName "kube-api-access-56xsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:50.684313 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:50.684268 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56xsr\" (UniqueName: \"kubernetes.io/projected/abc6bc4f-b6fe-49aa-9f79-41da164ffa08-kube-api-access-56xsr\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:50.684313 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:50.684315 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc6bc4f-b6fe-49aa-9f79-41da164ffa08-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:50.951653 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:50.951572 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:51.495173 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:51.495143 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:51.506948 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:51.506924 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw" Oct 30 21:51:51.507092 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:51.506965 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw" event={"ID":"abc6bc4f-b6fe-49aa-9f79-41da164ffa08","Type":"ContainerDied","Data":"5ed8747099fba5008b861ef66546e8ad60fb7bda61e62043a8f216e46b5b356a"} Oct 30 21:51:51.507092 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:51.507021 5801 scope.go:122] "RemoveContainer" containerID="14157862f3c04e54eb7387cdfb2527aaf664321bcb8403c105ce4962fd1d228d" Oct 30 21:51:51.524279 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:51.524250 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f421340-1852-46a4-9e66-4884e7c631f4" path="/var/lib/kubelet/pods/5f421340-1852-46a4-9e66-4884e7c631f4/volumes" Oct 30 21:51:51.534246 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:51.534211 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw"] Oct 30 21:51:51.539184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:51.539161 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1a81ae-7b7bcf5899-95nzw"] Oct 30 21:51:51.600191 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:51.600145 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:51.862910 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:51.862873 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz"] Oct 30 21:51:51.863153 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:51.863129 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" podUID="ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" containerName="istio-proxy" containerID="cri-o://cdfb2e6cbcd8969cd1624d0c0e07e081187aeb4b6ccb9b11f0ce73b91b9cc3db" gracePeriod=30 Oct 30 21:51:51.869973 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:51.869949 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:52.313624 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.313594 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8273be2-7k7tx"] Oct 30 21:51:52.314208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.314038 5801 cpu_manager.go:414] "containerMap: removing container" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" Oct 30 21:51:52.314208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.314057 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" Oct 30 21:51:52.314208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.314075 5801 cpu_manager.go:414] "containerMap: removing container" podUID="96f41b1e-ff7a-47b1-990a-677cdcd42a7f" containerName="istio-proxy" Oct 30 21:51:52.314208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.314084 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96f41b1e-ff7a-47b1-990a-677cdcd42a7f" containerName="istio-proxy" Oct 30 21:51:52.314208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.314131 5801 cpu_manager.go:414] "containerMap: removing container" podUID="abc6bc4f-b6fe-49aa-9f79-41da164ffa08" containerName="payload-processing" Oct 30 21:51:52.314208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.314140 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc6bc4f-b6fe-49aa-9f79-41da164ffa08" containerName="payload-processing" Oct 30 21:51:52.314208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.314146 5801 cpu_manager.go:414] "containerMap: removing container" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" Oct 30 21:51:52.314208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.314151 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" Oct 30 21:51:52.314208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.314165 5801 cpu_manager.go:414] "containerMap: removing container" podUID="fa0e622c-88d2-49b6-a7ea-8617987ee929" containerName="revoke-keys" Oct 30 21:51:52.314208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.314173 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa0e622c-88d2-49b6-a7ea-8617987ee929" containerName="revoke-keys" Oct 30 21:51:52.314208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.314183 5801 cpu_manager.go:414] "containerMap: removing container" podUID="457b1dc9-f94b-4dda-8b46-e125e0069033" containerName="istio-proxy" Oct 30 21:51:52.314208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.314189 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="457b1dc9-f94b-4dda-8b46-e125e0069033" containerName="istio-proxy" Oct 30 21:51:52.314208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.314198 5801 cpu_manager.go:414] "containerMap: removing container" podUID="457b1dc9-f94b-4dda-8b46-e125e0069033" containerName="istio-proxy" Oct 30 21:51:52.314208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.314202 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="457b1dc9-f94b-4dda-8b46-e125e0069033" containerName="istio-proxy" Oct 30 21:51:52.314712 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.314275 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc6bc4f-b6fe-49aa-9f79-41da164ffa08" containerName="payload-processing" Oct 30 21:51:52.314712 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.314283 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="363ebad1-96ba-4cf4-b0b9-2a93cbf1bbdb" containerName="main" Oct 30 21:51:52.314712 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.314291 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa0e622c-88d2-49b6-a7ea-8617987ee929" containerName="revoke-keys" Oct 30 21:51:52.314712 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.314296 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="457b1dc9-f94b-4dda-8b46-e125e0069033" containerName="istio-proxy" Oct 30 21:51:52.314712 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.314302 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="457b1dc9-f94b-4dda-8b46-e125e0069033" containerName="istio-proxy" Oct 30 21:51:52.314712 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.314308 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea440bed-5030-4eeb-85b7-1916ffe802af" containerName="main" Oct 30 21:51:52.314712 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.314314 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="96f41b1e-ff7a-47b1-990a-677cdcd42a7f" containerName="istio-proxy" Oct 30 21:51:52.318807 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.318786 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8273be2-7k7tx" Oct 30 21:51:52.327344 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.327298 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:51:52.327522 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.327507 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 21:51:52.329816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.329797 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8273be2-7k7tx"] Oct 30 21:51:52.374895 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.374874 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:51:52.374895 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.374875 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:51:52.402384 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.402358 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d7e580-7045-4135-beaf-03297571ed5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8273be2-7k7tx\" (UID: \"01d7e580-7045-4135-beaf-03297571ed5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8273be2-7k7tx" Oct 30 21:51:52.402493 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.402390 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d7e580-7045-4135-beaf-03297571ed5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8273be2-7k7tx\" (UID: \"01d7e580-7045-4135-beaf-03297571ed5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8273be2-7k7tx" Oct 30 21:51:52.402493 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.402411 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mv9m\" (UniqueName: \"kubernetes.io/projected/01d7e580-7045-4135-beaf-03297571ed5a-kube-api-access-7mv9m\") pod \"maas-api-revoke-sub-a8273be2-7k7tx\" (UID: \"01d7e580-7045-4135-beaf-03297571ed5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8273be2-7k7tx" Oct 30 21:51:52.503585 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.503559 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d7e580-7045-4135-beaf-03297571ed5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8273be2-7k7tx\" (UID: \"01d7e580-7045-4135-beaf-03297571ed5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8273be2-7k7tx" Oct 30 21:51:52.503989 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.503593 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d7e580-7045-4135-beaf-03297571ed5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8273be2-7k7tx\" (UID: \"01d7e580-7045-4135-beaf-03297571ed5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8273be2-7k7tx" Oct 30 21:51:52.503989 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.503615 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mv9m\" (UniqueName: \"kubernetes.io/projected/01d7e580-7045-4135-beaf-03297571ed5a-kube-api-access-7mv9m\") pod \"maas-api-revoke-sub-a8273be2-7k7tx\" (UID: \"01d7e580-7045-4135-beaf-03297571ed5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8273be2-7k7tx" Oct 30 21:51:52.504321 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.504299 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d7e580-7045-4135-beaf-03297571ed5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8273be2-7k7tx\" (UID: \"01d7e580-7045-4135-beaf-03297571ed5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8273be2-7k7tx" Oct 30 21:51:52.505962 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.505942 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d7e580-7045-4135-beaf-03297571ed5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8273be2-7k7tx\" (UID: \"01d7e580-7045-4135-beaf-03297571ed5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8273be2-7k7tx" Oct 30 21:51:52.519065 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.519037 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mv9m\" (UniqueName: \"kubernetes.io/projected/01d7e580-7045-4135-beaf-03297571ed5a-kube-api-access-7mv9m\") pod \"maas-api-revoke-sub-a8273be2-7k7tx\" (UID: \"01d7e580-7045-4135-beaf-03297571ed5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8273be2-7k7tx" Oct 30 21:51:52.633636 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.633583 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8273be2-7k7tx" Oct 30 21:51:52.920842 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:52.920816 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8273be2-7k7tx"] Oct 30 21:51:52.923050 ip-10-0-140-163 kubenswrapper[5801]: W1030 21:51:52.923022 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d7e580_7045_4135_beaf_03297571ed5a.slice/crio-d101e7bb46e0811b7ab28bd9d967c14a0afe53459c95ed3e74c2196272581652 WatchSource:0}: Error finding container d101e7bb46e0811b7ab28bd9d967c14a0afe53459c95ed3e74c2196272581652: Status 404 returned error can't find the container with id d101e7bb46e0811b7ab28bd9d967c14a0afe53459c95ed3e74c2196272581652 Oct 30 21:51:53.497270 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.497239 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:51:53.539240 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.539204 5801 generic.go:356] "Generic (PLEG): container finished" podID="0f258bc9-d2fd-446a-9154-36caf951e081" containerID="91184ee5cdc7e7d95c441873ef8ab2d8bbe6cf53bd147cd47d8682d422463155" exitCode=0 Oct 30 21:51:53.539976 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.539953 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" Oct 30 21:51:53.545519 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.545494 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc6bc4f-b6fe-49aa-9f79-41da164ffa08" path="/var/lib/kubelet/pods/abc6bc4f-b6fe-49aa-9f79-41da164ffa08/volumes" Oct 30 21:51:53.548162 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.548140 5801 generic.go:356] "Generic (PLEG): container finished" podID="9a837073-7227-4ae9-9338-a70202213538" containerID="bd461a09ddfde0ed4c2332971e30b94c12f1ecb322523dd7ad8d581e0ad2cbc3" exitCode=0 Oct 30 21:51:53.561636 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.561612 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" event={"ID":"0f258bc9-d2fd-446a-9154-36caf951e081","Type":"ContainerDied","Data":"91184ee5cdc7e7d95c441873ef8ab2d8bbe6cf53bd147cd47d8682d422463155"} Oct 30 21:51:53.561789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.561669 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl" event={"ID":"0f258bc9-d2fd-446a-9154-36caf951e081","Type":"ContainerDied","Data":"0cef7a4ed47262749f6760a23375cfae663aeeae2988aae6c4a8c66fcb9e5407"} Oct 30 21:51:53.561789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.561688 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8273be2-7k7tx" event={"ID":"01d7e580-7045-4135-beaf-03297571ed5a","Type":"ContainerStarted","Data":"2366b3748b4981a600ad55faf9c3238b7f8f6d966f368a6856aec800393a1266"} Oct 30 21:51:53.561789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.561703 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8273be2-7k7tx" event={"ID":"01d7e580-7045-4135-beaf-03297571ed5a","Type":"ContainerStarted","Data":"d101e7bb46e0811b7ab28bd9d967c14a0afe53459c95ed3e74c2196272581652"} Oct 30 21:51:53.561789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.561721 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" event={"ID":"9a837073-7227-4ae9-9338-a70202213538","Type":"ContainerDied","Data":"bd461a09ddfde0ed4c2332971e30b94c12f1ecb322523dd7ad8d581e0ad2cbc3"} Oct 30 21:51:53.561789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.561738 5801 scope.go:122] "RemoveContainer" containerID="91184ee5cdc7e7d95c441873ef8ab2d8bbe6cf53bd147cd47d8682d422463155" Oct 30 21:51:53.568041 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.567997 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8273be2-7k7tx" podStartSLOduration=1.5679843020000002 podStartE2EDuration="1.567984302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:53.564717413 +0000 UTC m=+1382.160534085" watchObservedRunningTime="2026-10-30 21:51:53.567984302 +0000 UTC m=+1382.163800966" Oct 30 21:51:53.569566 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.569542 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:51:53.573307 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.573284 5801 scope.go:122] "RemoveContainer" containerID="76fe2b8ce6f3ea3a2d1c7ecec5e9b7bd17b1b9a550a56410f745684e91c3ce96" Oct 30 21:51:53.582631 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.582607 5801 scope.go:122] "RemoveContainer" containerID="91184ee5cdc7e7d95c441873ef8ab2d8bbe6cf53bd147cd47d8682d422463155" Oct 30 21:51:53.582982 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:53.582962 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91184ee5cdc7e7d95c441873ef8ab2d8bbe6cf53bd147cd47d8682d422463155\": container with ID starting with 91184ee5cdc7e7d95c441873ef8ab2d8bbe6cf53bd147cd47d8682d422463155 not found: ID does not exist" containerID="91184ee5cdc7e7d95c441873ef8ab2d8bbe6cf53bd147cd47d8682d422463155" Oct 30 21:51:53.583032 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.582990 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91184ee5cdc7e7d95c441873ef8ab2d8bbe6cf53bd147cd47d8682d422463155"} err="failed to get container status \"91184ee5cdc7e7d95c441873ef8ab2d8bbe6cf53bd147cd47d8682d422463155\": rpc error: code = NotFound desc = could not find container \"91184ee5cdc7e7d95c441873ef8ab2d8bbe6cf53bd147cd47d8682d422463155\": container with ID starting with 91184ee5cdc7e7d95c441873ef8ab2d8bbe6cf53bd147cd47d8682d422463155 not found: ID does not exist" Oct 30 21:51:53.583032 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.583007 5801 scope.go:122] "RemoveContainer" containerID="76fe2b8ce6f3ea3a2d1c7ecec5e9b7bd17b1b9a550a56410f745684e91c3ce96" Oct 30 21:51:53.583243 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:53.583224 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76fe2b8ce6f3ea3a2d1c7ecec5e9b7bd17b1b9a550a56410f745684e91c3ce96\": container with ID starting with 76fe2b8ce6f3ea3a2d1c7ecec5e9b7bd17b1b9a550a56410f745684e91c3ce96 not found: ID does not exist" containerID="76fe2b8ce6f3ea3a2d1c7ecec5e9b7bd17b1b9a550a56410f745684e91c3ce96" Oct 30 21:51:53.583281 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.583249 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76fe2b8ce6f3ea3a2d1c7ecec5e9b7bd17b1b9a550a56410f745684e91c3ce96"} err="failed to get container status \"76fe2b8ce6f3ea3a2d1c7ecec5e9b7bd17b1b9a550a56410f745684e91c3ce96\": rpc error: code = NotFound desc = could not find container \"76fe2b8ce6f3ea3a2d1c7ecec5e9b7bd17b1b9a550a56410f745684e91c3ce96\": container with ID starting with 76fe2b8ce6f3ea3a2d1c7ecec5e9b7bd17b1b9a550a56410f745684e91c3ce96 not found: ID does not exist" Oct 30 21:51:53.616391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.616355 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-workload-certs\") pod \"0f258bc9-d2fd-446a-9154-36caf951e081\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " Oct 30 21:51:53.616391 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.616390 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a837073-7227-4ae9-9338-a70202213538-istio-token\" (UniqueName: \"kubernetes.io/projected/9a837073-7227-4ae9-9338-a70202213538-istio-token\") pod \"9a837073-7227-4ae9-9338-a70202213538\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " Oct 30 21:51:53.616569 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.616441 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-istio-envoy\") pod \"0f258bc9-d2fd-446a-9154-36caf951e081\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " Oct 30 21:51:53.616569 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.616470 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-istio-data\") pod \"9a837073-7227-4ae9-9338-a70202213538\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " Oct 30 21:51:53.616569 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.616496 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f258bc9-d2fd-446a-9154-36caf951e081-kube-api-access-c9c4s\" (UniqueName: \"kubernetes.io/projected/0f258bc9-d2fd-446a-9154-36caf951e081-kube-api-access-c9c4s\") pod \"0f258bc9-d2fd-446a-9154-36caf951e081\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " Oct 30 21:51:53.616569 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.616524 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-credential-socket\") pod \"9a837073-7227-4ae9-9338-a70202213538\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " Oct 30 21:51:53.616853 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.616572 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-workload-socket\") pod \"9a837073-7227-4ae9-9338-a70202213538\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " Oct 30 21:51:53.616853 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.616629 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-istio-envoy\") pod \"9a837073-7227-4ae9-9338-a70202213538\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " Oct 30 21:51:53.616853 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.616641 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-workload-certs" pod "0f258bc9-d2fd-446a-9154-36caf951e081" (UID: "0f258bc9-d2fd-446a-9154-36caf951e081"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:53.616853 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.616667 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f258bc9-d2fd-446a-9154-36caf951e081-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f258bc9-d2fd-446a-9154-36caf951e081-istio-podinfo\") pod \"0f258bc9-d2fd-446a-9154-36caf951e081\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " Oct 30 21:51:53.616853 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.616695 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a837073-7227-4ae9-9338-a70202213538-kube-api-access-wnbxs\" (UniqueName: \"kubernetes.io/projected/9a837073-7227-4ae9-9338-a70202213538-kube-api-access-wnbxs\") pod \"9a837073-7227-4ae9-9338-a70202213538\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " Oct 30 21:51:53.616853 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.616743 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f258bc9-d2fd-446a-9154-36caf951e081-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f258bc9-d2fd-446a-9154-36caf951e081-istiod-ca-cert\") pod \"0f258bc9-d2fd-446a-9154-36caf951e081\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " Oct 30 21:51:53.616853 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.616798 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a837073-7227-4ae9-9338-a70202213538-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a837073-7227-4ae9-9338-a70202213538-istiod-ca-cert\") pod \"9a837073-7227-4ae9-9338-a70202213538\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " Oct 30 21:51:53.616853 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.616828 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-istio-data\") pod \"0f258bc9-d2fd-446a-9154-36caf951e081\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " Oct 30 21:51:53.617279 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.616874 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-credential-socket\") pod \"0f258bc9-d2fd-446a-9154-36caf951e081\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " Oct 30 21:51:53.617279 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.616869 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-istio-data" pod "9a837073-7227-4ae9-9338-a70202213538" (UID: "9a837073-7227-4ae9-9338-a70202213538"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:53.617279 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.616905 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-credential-socket" pod "9a837073-7227-4ae9-9338-a70202213538" (UID: "9a837073-7227-4ae9-9338-a70202213538"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:53.617279 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.616925 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-workload-certs\") pod \"9a837073-7227-4ae9-9338-a70202213538\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " Oct 30 21:51:53.617279 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.617015 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f258bc9-d2fd-446a-9154-36caf951e081-istio-token\" (UniqueName: \"kubernetes.io/projected/0f258bc9-d2fd-446a-9154-36caf951e081-istio-token\") pod \"0f258bc9-d2fd-446a-9154-36caf951e081\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " Oct 30 21:51:53.617279 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.617050 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-workload-socket\") pod \"0f258bc9-d2fd-446a-9154-36caf951e081\" (UID: \"0f258bc9-d2fd-446a-9154-36caf951e081\") " Oct 30 21:51:53.617279 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.617080 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a837073-7227-4ae9-9338-a70202213538-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a837073-7227-4ae9-9338-a70202213538-istio-podinfo\") pod \"9a837073-7227-4ae9-9338-a70202213538\" (UID: \"9a837073-7227-4ae9-9338-a70202213538\") " Oct 30 21:51:53.617655 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.617421 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-credential-socket" pod "0f258bc9-d2fd-446a-9154-36caf951e081" (UID: "0f258bc9-d2fd-446a-9154-36caf951e081"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:53.617655 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.617645 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a837073-7227-4ae9-9338-a70202213538-istiod-ca-cert" pod "9a837073-7227-4ae9-9338-a70202213538" (UID: "9a837073-7227-4ae9-9338-a70202213538"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:53.617776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.617673 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f258bc9-d2fd-446a-9154-36caf951e081-istiod-ca-cert" pod "0f258bc9-d2fd-446a-9154-36caf951e081" (UID: "0f258bc9-d2fd-446a-9154-36caf951e081"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:53.618292 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.618158 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-workload-socket" pod "9a837073-7227-4ae9-9338-a70202213538" (UID: "9a837073-7227-4ae9-9338-a70202213538"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:53.618408 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.618316 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-istio-data" pod "0f258bc9-d2fd-446a-9154-36caf951e081" (UID: "0f258bc9-d2fd-446a-9154-36caf951e081"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:53.618931 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.618905 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-workload-certs" pod "9a837073-7227-4ae9-9338-a70202213538" (UID: "9a837073-7227-4ae9-9338-a70202213538"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:53.619091 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.619070 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-workload-socket" pod "0f258bc9-d2fd-446a-9154-36caf951e081" (UID: "0f258bc9-d2fd-446a-9154-36caf951e081"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:53.619181 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.619138 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-istio-envoy" pod "0f258bc9-d2fd-446a-9154-36caf951e081" (UID: "0f258bc9-d2fd-446a-9154-36caf951e081"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:53.619599 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.619577 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.619686 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.619604 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.619686 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.619620 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.619686 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.619635 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.619686 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.619649 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.619686 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.619663 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.619686 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.619678 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f258bc9-d2fd-446a-9154-36caf951e081-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.620053 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.619694 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a837073-7227-4ae9-9338-a70202213538-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.620053 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.619708 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.620053 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.619721 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f258bc9-d2fd-446a-9154-36caf951e081-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.620053 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.619735 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.620053 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.619870 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f258bc9-d2fd-446a-9154-36caf951e081-kube-api-access-c9c4s" pod "0f258bc9-d2fd-446a-9154-36caf951e081" (UID: "0f258bc9-d2fd-446a-9154-36caf951e081"). InnerVolumeSpecName "kube-api-access-c9c4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:53.620053 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.619897 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a837073-7227-4ae9-9338-a70202213538-istio-token" pod "9a837073-7227-4ae9-9338-a70202213538" (UID: "9a837073-7227-4ae9-9338-a70202213538"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:53.620365 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.620214 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-istio-envoy" pod "9a837073-7227-4ae9-9338-a70202213538" (UID: "9a837073-7227-4ae9-9338-a70202213538"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:53.620846 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.620820 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a837073-7227-4ae9-9338-a70202213538-istio-podinfo" pod "9a837073-7227-4ae9-9338-a70202213538" (UID: "9a837073-7227-4ae9-9338-a70202213538"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:53.621219 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.621198 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a837073-7227-4ae9-9338-a70202213538-kube-api-access-wnbxs" pod "9a837073-7227-4ae9-9338-a70202213538" (UID: "9a837073-7227-4ae9-9338-a70202213538"). InnerVolumeSpecName "kube-api-access-wnbxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:53.621624 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.621605 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f258bc9-d2fd-446a-9154-36caf951e081-istio-podinfo" pod "0f258bc9-d2fd-446a-9154-36caf951e081" (UID: "0f258bc9-d2fd-446a-9154-36caf951e081"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:53.621742 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.621725 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f258bc9-d2fd-446a-9154-36caf951e081-istio-token" pod "0f258bc9-d2fd-446a-9154-36caf951e081" (UID: "0f258bc9-d2fd-446a-9154-36caf951e081"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:53.721153 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.721118 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a837073-7227-4ae9-9338-a70202213538-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.721153 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.721149 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f258bc9-d2fd-446a-9154-36caf951e081-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.721340 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.721165 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnbxs\" (UniqueName: \"kubernetes.io/projected/9a837073-7227-4ae9-9338-a70202213538-kube-api-access-wnbxs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.721340 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.721179 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f258bc9-d2fd-446a-9154-36caf951e081-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.721340 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.721191 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a837073-7227-4ae9-9338-a70202213538-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.721340 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.721203 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a837073-7227-4ae9-9338-a70202213538-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.721340 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.721216 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9c4s\" (UniqueName: \"kubernetes.io/projected/0f258bc9-d2fd-446a-9154-36caf951e081-kube-api-access-c9c4s\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.863868 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.863840 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl"] Oct 30 21:51:53.867875 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:53.867850 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-275jl"] Oct 30 21:51:54.559319 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:54.559281 5801 generic.go:356] "Generic (PLEG): container finished" podID="01d7e580-7045-4135-beaf-03297571ed5a" containerID="2366b3748b4981a600ad55faf9c3238b7f8f6d966f368a6856aec800393a1266" exitCode=0 Oct 30 21:51:54.559657 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:54.559365 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8273be2-7k7tx" event={"ID":"01d7e580-7045-4135-beaf-03297571ed5a","Type":"ContainerDied","Data":"2366b3748b4981a600ad55faf9c3238b7f8f6d966f368a6856aec800393a1266"} Oct 30 21:51:54.560799 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:54.560773 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" event={"ID":"9a837073-7227-4ae9-9338-a70202213538","Type":"ContainerDied","Data":"4346bf01dca7c0cccd45ad822bf394bfc25ef10471a21e9d5c3b3e7dc20a8665"} Oct 30 21:51:54.560905 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:54.560799 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd" Oct 30 21:51:54.560905 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:54.560820 5801 scope.go:122] "RemoveContainer" containerID="bd461a09ddfde0ed4c2332971e30b94c12f1ecb322523dd7ad8d581e0ad2cbc3" Oct 30 21:51:54.592363 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:54.592339 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd"] Oct 30 21:51:54.600448 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:54.600427 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0fa864-openshift-default-97b498755-wncrd"] Oct 30 21:51:55.523173 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:55.523141 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f258bc9-d2fd-446a-9154-36caf951e081" path="/var/lib/kubelet/pods/0f258bc9-d2fd-446a-9154-36caf951e081/volumes" Oct 30 21:51:55.523591 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:55.523578 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a837073-7227-4ae9-9338-a70202213538" path="/var/lib/kubelet/pods/9a837073-7227-4ae9-9338-a70202213538/volumes" Oct 30 21:51:55.705294 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:55.705268 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8273be2-7k7tx" Oct 30 21:51:55.845441 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:55.845414 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01d7e580-7045-4135-beaf-03297571ed5a-kube-api-access-7mv9m\" (UniqueName: \"kubernetes.io/projected/01d7e580-7045-4135-beaf-03297571ed5a-kube-api-access-7mv9m\") pod \"01d7e580-7045-4135-beaf-03297571ed5a\" (UID: \"01d7e580-7045-4135-beaf-03297571ed5a\") " Oct 30 21:51:55.845595 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:55.845536 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01d7e580-7045-4135-beaf-03297571ed5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d7e580-7045-4135-beaf-03297571ed5a-maas-api-xks-ca\") pod \"01d7e580-7045-4135-beaf-03297571ed5a\" (UID: \"01d7e580-7045-4135-beaf-03297571ed5a\") " Oct 30 21:51:55.845595 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:55.845562 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01d7e580-7045-4135-beaf-03297571ed5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d7e580-7045-4135-beaf-03297571ed5a-maas-api-openshift-ca\") pod \"01d7e580-7045-4135-beaf-03297571ed5a\" (UID: \"01d7e580-7045-4135-beaf-03297571ed5a\") " Oct 30 21:51:55.846027 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:55.846003 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01d7e580-7045-4135-beaf-03297571ed5a-maas-api-openshift-ca" pod "01d7e580-7045-4135-beaf-03297571ed5a" (UID: "01d7e580-7045-4135-beaf-03297571ed5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:55.847438 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:55.847414 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01d7e580-7045-4135-beaf-03297571ed5a-maas-api-xks-ca" pod "01d7e580-7045-4135-beaf-03297571ed5a" (UID: "01d7e580-7045-4135-beaf-03297571ed5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:55.847531 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:55.847448 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d7e580-7045-4135-beaf-03297571ed5a-kube-api-access-7mv9m" pod "01d7e580-7045-4135-beaf-03297571ed5a" (UID: "01d7e580-7045-4135-beaf-03297571ed5a"). InnerVolumeSpecName "kube-api-access-7mv9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:55.878332 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:55.878306 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:55.946673 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:55.946650 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d7e580-7045-4135-beaf-03297571ed5a-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:55.946673 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:55.946674 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d7e580-7045-4135-beaf-03297571ed5a-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:55.946829 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:55.946686 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mv9m\" (UniqueName: \"kubernetes.io/projected/01d7e580-7045-4135-beaf-03297571ed5a-kube-api-access-7mv9m\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:56.577767 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:56.577735 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8273be2-7k7tx" Oct 30 21:51:56.577952 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:56.577782 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8273be2-7k7tx" event={"ID":"01d7e580-7045-4135-beaf-03297571ed5a","Type":"ContainerDied","Data":"d101e7bb46e0811b7ab28bd9d967c14a0afe53459c95ed3e74c2196272581652"} Oct 30 21:51:56.577952 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:56.577822 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d101e7bb46e0811b7ab28bd9d967c14a0afe53459c95ed3e74c2196272581652" Oct 30 21:51:57.069262 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.069198 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:57.143902 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.143877 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:51:57.177242 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.177207 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:57.268005 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.267969 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-envoy\") pod \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " Oct 30 21:51:57.268005 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.268007 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istiod-ca-cert\") pod \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " Oct 30 21:51:57.268220 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.268026 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-workload-socket\") pod \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " Oct 30 21:51:57.268220 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.268052 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-podinfo\") pod \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " Oct 30 21:51:57.268331 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.268220 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-workload-certs\") pod \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " Oct 30 21:51:57.268331 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.268279 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-data\") pod \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " Oct 30 21:51:57.268476 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.268305 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-workload-socket" pod "ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" (UID: "ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:57.268476 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.268369 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-credential-socket\") pod \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " Oct 30 21:51:57.268476 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.268437 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-token\" (UniqueName: \"kubernetes.io/projected/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-token\") pod \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " Oct 30 21:51:57.268476 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.268456 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istiod-ca-cert" pod "ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" (UID: "ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:57.268476 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.268468 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-kube-api-access-42crw\" (UniqueName: \"kubernetes.io/projected/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-kube-api-access-42crw\") pod \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\" (UID: \"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2\") " Oct 30 21:51:57.268749 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.268601 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-workload-certs" pod "ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" (UID: "ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:57.268842 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.268799 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-credential-socket" pod "ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" (UID: "ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:57.268960 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.268936 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.268960 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.268949 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-data" pod "ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" (UID: "ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:57.268960 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.268966 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.269173 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.268983 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.269173 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.268997 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.270347 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.270314 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-envoy" pod "ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" (UID: "ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:57.270777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.270744 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-token" pod "ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" (UID: "ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:57.270877 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.270739 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-podinfo" pod "ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" (UID: "ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:57.270976 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.270952 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-kube-api-access-42crw" pod "ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" (UID: "ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2"). InnerVolumeSpecName "kube-api-access-42crw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:57.370380 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.370339 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.370380 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.370376 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.370653 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.370391 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.370653 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.370404 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.370653 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.370419 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42crw\" (UniqueName: \"kubernetes.io/projected/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2-kube-api-access-42crw\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.586723 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.586694 5801 generic.go:356] "Generic (PLEG): container finished" podID="ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" containerID="cdfb2e6cbcd8969cd1624d0c0e07e081187aeb4b6ccb9b11f0ce73b91b9cc3db" exitCode=0 Oct 30 21:51:57.586913 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.586740 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" Oct 30 21:51:57.586913 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.586782 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" event={"ID":"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2","Type":"ContainerDied","Data":"cdfb2e6cbcd8969cd1624d0c0e07e081187aeb4b6ccb9b11f0ce73b91b9cc3db"} Oct 30 21:51:57.586913 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.586832 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz" event={"ID":"ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2","Type":"ContainerDied","Data":"9dd412d8651bfb13ad9d03222019cb9951d8b49e138dc2e9cc4597376621dd2e"} Oct 30 21:51:57.586913 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.586853 5801 scope.go:122] "RemoveContainer" containerID="cdfb2e6cbcd8969cd1624d0c0e07e081187aeb4b6ccb9b11f0ce73b91b9cc3db" Oct 30 21:51:57.597538 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.597518 5801 scope.go:122] "RemoveContainer" containerID="30c565eb326a7c2d89560e7c9e91fd808763b066617b64f4335aa365f6f103b5" Oct 30 21:51:57.610504 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.610484 5801 scope.go:122] "RemoveContainer" containerID="cdfb2e6cbcd8969cd1624d0c0e07e081187aeb4b6ccb9b11f0ce73b91b9cc3db" Oct 30 21:51:57.611899 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:57.611873 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdfb2e6cbcd8969cd1624d0c0e07e081187aeb4b6ccb9b11f0ce73b91b9cc3db\": container with ID starting with cdfb2e6cbcd8969cd1624d0c0e07e081187aeb4b6ccb9b11f0ce73b91b9cc3db not found: ID does not exist" containerID="cdfb2e6cbcd8969cd1624d0c0e07e081187aeb4b6ccb9b11f0ce73b91b9cc3db" Oct 30 21:51:57.612000 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.611910 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdfb2e6cbcd8969cd1624d0c0e07e081187aeb4b6ccb9b11f0ce73b91b9cc3db"} err="failed to get container status \"cdfb2e6cbcd8969cd1624d0c0e07e081187aeb4b6ccb9b11f0ce73b91b9cc3db\": rpc error: code = NotFound desc = could not find container \"cdfb2e6cbcd8969cd1624d0c0e07e081187aeb4b6ccb9b11f0ce73b91b9cc3db\": container with ID starting with cdfb2e6cbcd8969cd1624d0c0e07e081187aeb4b6ccb9b11f0ce73b91b9cc3db not found: ID does not exist" Oct 30 21:51:57.612000 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.611940 5801 scope.go:122] "RemoveContainer" containerID="30c565eb326a7c2d89560e7c9e91fd808763b066617b64f4335aa365f6f103b5" Oct 30 21:51:57.612211 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:51:57.612193 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30c565eb326a7c2d89560e7c9e91fd808763b066617b64f4335aa365f6f103b5\": container with ID starting with 30c565eb326a7c2d89560e7c9e91fd808763b066617b64f4335aa365f6f103b5 not found: ID does not exist" containerID="30c565eb326a7c2d89560e7c9e91fd808763b066617b64f4335aa365f6f103b5" Oct 30 21:51:57.612251 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.612217 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30c565eb326a7c2d89560e7c9e91fd808763b066617b64f4335aa365f6f103b5"} err="failed to get container status \"30c565eb326a7c2d89560e7c9e91fd808763b066617b64f4335aa365f6f103b5\": rpc error: code = NotFound desc = could not find container \"30c565eb326a7c2d89560e7c9e91fd808763b066617b64f4335aa365f6f103b5\": container with ID starting with 30c565eb326a7c2d89560e7c9e91fd808763b066617b64f4335aa365f6f103b5 not found: ID does not exist" Oct 30 21:51:57.634313 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.634289 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz"] Oct 30 21:51:57.641106 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:57.641081 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1a81ae-openshift-default-cc77b7797-fz2sz"] Oct 30 21:51:58.088708 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:58.088675 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:58.224695 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:58.224646 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:58.226785 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:58.226734 5801 scope.go:122] "RemoveContainer" containerID="d1e0e9c07264e74f7b06ba9721ad499bc6796c1114a8fd5e28d34f3ad736d1bf" Oct 30 21:51:58.248278 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:58.248253 5801 scope.go:122] "RemoveContainer" containerID="608ce07e38703cb3970537f001c8a68995e9500e91a9a0162ebe998874cfabc1" Oct 30 21:51:58.407611 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:58.407529 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:58.563035 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:58.562707 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:58.693777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:58.689974 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:59.045869 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:59.045780 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:59.149217 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:59.148231 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:59.528267 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:59.528237 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" path="/var/lib/kubelet/pods/ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2/volumes" Oct 30 21:51:59.592618 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:59.592586 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5"] Oct 30 21:51:59.592930 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:59.592901 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" podUID="1fcdad2f-d069-41a7-ace4-49d0ba4abfaa" containerName="istio-proxy" containerID="cri-o://6bbadfdda2e87646a87cad137968f08f6b0e9b7972e31f1fc58d65c2eb5f43f0" gracePeriod=30 Oct 30 21:51:59.597217 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:59.595723 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f"] Oct 30 21:51:59.597217 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:59.595971 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" podUID="e3976e0e-4921-46d0-99f4-2c26733ef15a" containerName="istio-proxy" containerID="cri-o://62dc366945d2c550b3e888b0e7d4b2ac39efa9bd3f4952e4310d1eb5ad2daa04" gracePeriod=30 Oct 30 21:51:59.615200 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:59.615169 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:59.857085 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:59.857051 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:59.930709 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:59.930674 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" containerID="cri-o://fe597c9e61aa17655ab23710562b507ead65ae66da9d138535e51044324e8375" gracePeriod=45 Oct 30 21:51:59.941228 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:51:59.941190 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" containerID="cri-o://b7e7d3188cd510e938bf4835623a2034161095aa47f6b6e80f55a65db60776c4" gracePeriod=45 Oct 30 21:52:00.187861 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.187838 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:52:00.205030 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.205004 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:52:00.214259 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.214238 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d49a00ed-7af8-4999-91d7-ee8174080727-kube-api-access-dzwf6\" (UniqueName: \"kubernetes.io/projected/d49a00ed-7af8-4999-91d7-ee8174080727-kube-api-access-dzwf6\") pod \"d49a00ed-7af8-4999-91d7-ee8174080727\" (UID: \"d49a00ed-7af8-4999-91d7-ee8174080727\") " Oct 30 21:52:00.214320 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.214269 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-tmp-dir\") pod \"d49a00ed-7af8-4999-91d7-ee8174080727\" (UID: \"d49a00ed-7af8-4999-91d7-ee8174080727\") " Oct 30 21:52:00.214320 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.214305 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-dshm\") pod \"d49a00ed-7af8-4999-91d7-ee8174080727\" (UID: \"d49a00ed-7af8-4999-91d7-ee8174080727\") " Oct 30 21:52:00.214393 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.214382 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-home\" (UniqueName: \"kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-home\") pod \"d49a00ed-7af8-4999-91d7-ee8174080727\" (UID: \"d49a00ed-7af8-4999-91d7-ee8174080727\") " Oct 30 21:52:00.214445 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.214430 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-model-cache\") pod \"d49a00ed-7af8-4999-91d7-ee8174080727\" (UID: \"d49a00ed-7af8-4999-91d7-ee8174080727\") " Oct 30 21:52:00.214492 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.214467 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d49a00ed-7af8-4999-91d7-ee8174080727-tls-certs\" (UniqueName: \"kubernetes.io/secret/d49a00ed-7af8-4999-91d7-ee8174080727-tls-certs\") pod \"d49a00ed-7af8-4999-91d7-ee8174080727\" (UID: \"d49a00ed-7af8-4999-91d7-ee8174080727\") " Oct 30 21:52:00.214542 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.214512 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-tmp-dir" pod "d49a00ed-7af8-4999-91d7-ee8174080727" (UID: "d49a00ed-7af8-4999-91d7-ee8174080727"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:00.214691 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.214669 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-home" pod "d49a00ed-7af8-4999-91d7-ee8174080727" (UID: "d49a00ed-7af8-4999-91d7-ee8174080727"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:00.214841 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.214688 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-model-cache" pod "d49a00ed-7af8-4999-91d7-ee8174080727" (UID: "d49a00ed-7af8-4999-91d7-ee8174080727"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:00.214975 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.214957 5801 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:00.215041 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.214980 5801 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:00.215041 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.214996 5801 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:00.216552 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.216526 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-dshm" pod "d49a00ed-7af8-4999-91d7-ee8174080727" (UID: "d49a00ed-7af8-4999-91d7-ee8174080727"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:00.216651 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.216577 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49a00ed-7af8-4999-91d7-ee8174080727-kube-api-access-dzwf6" pod "d49a00ed-7af8-4999-91d7-ee8174080727" (UID: "d49a00ed-7af8-4999-91d7-ee8174080727"). InnerVolumeSpecName "kube-api-access-dzwf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:00.216651 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.216576 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d49a00ed-7af8-4999-91d7-ee8174080727-tls-certs" pod "d49a00ed-7af8-4999-91d7-ee8174080727" (UID: "d49a00ed-7af8-4999-91d7-ee8174080727"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:52:00.286490 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:52:00.286447 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:52:00.316074 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.316045 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-tmp-dir\") pod \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\" (UID: \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\") " Oct 30 21:52:00.316246 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.316119 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-model-cache\") pod \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\" (UID: \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\") " Oct 30 21:52:00.316246 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.316148 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-tls-certs\" (UniqueName: \"kubernetes.io/secret/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-tls-certs\") pod \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\" (UID: \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\") " Oct 30 21:52:00.316246 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.316170 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-kube-api-access-dhdmd\" (UniqueName: \"kubernetes.io/projected/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-kube-api-access-dhdmd\") pod \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\" (UID: \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\") " Oct 30 21:52:00.316434 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.316262 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-dshm\") pod \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\" (UID: \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\") " Oct 30 21:52:00.316434 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.316289 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-home\" (UniqueName: \"kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-home\") pod \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\" (UID: \"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1\") " Oct 30 21:52:00.316434 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.316396 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-tmp-dir" pod "bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" (UID: "bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:00.316610 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.316445 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-model-cache" pod "bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" (UID: "bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:00.316679 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.316606 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-home" pod "bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" (UID: "bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:00.316795 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.316777 5801 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:00.316866 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.316800 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzwf6\" (UniqueName: \"kubernetes.io/projected/d49a00ed-7af8-4999-91d7-ee8174080727-kube-api-access-dzwf6\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:00.316866 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.316817 5801 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d49a00ed-7af8-4999-91d7-ee8174080727-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:00.316866 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.316830 5801 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:00.316866 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.316846 5801 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d49a00ed-7af8-4999-91d7-ee8174080727-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:00.316866 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.316858 5801 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:00.318387 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.318360 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-kube-api-access-dhdmd" pod "bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" (UID: "bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1"). InnerVolumeSpecName "kube-api-access-dhdmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:00.318387 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.318375 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-tls-certs" pod "bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" (UID: "bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:52:00.318551 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.318423 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-dshm" pod "bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" (UID: "bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:00.418135 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.418069 5801 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:00.418135 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.418094 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhdmd\" (UniqueName: \"kubernetes.io/projected/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-kube-api-access-dhdmd\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:00.418135 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.418104 5801 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:00.651561 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.651531 5801 generic.go:356] "Generic (PLEG): container finished" podID="d49a00ed-7af8-4999-91d7-ee8174080727" containerID="fe597c9e61aa17655ab23710562b507ead65ae66da9d138535e51044324e8375" exitCode=0 Oct 30 21:52:00.651793 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.651573 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" Oct 30 21:52:00.651793 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.651624 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" event={"ID":"d49a00ed-7af8-4999-91d7-ee8174080727","Type":"ContainerDied","Data":"fe597c9e61aa17655ab23710562b507ead65ae66da9d138535e51044324e8375"} Oct 30 21:52:00.651793 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.651664 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p" event={"ID":"d49a00ed-7af8-4999-91d7-ee8174080727","Type":"ContainerDied","Data":"3bdf5e0a11abac49a964863adef928070331c8112b67a08469bfdcf93c4a1f0b"} Oct 30 21:52:00.651793 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.651686 5801 scope.go:122] "RemoveContainer" containerID="fe597c9e61aa17655ab23710562b507ead65ae66da9d138535e51044324e8375" Oct 30 21:52:00.654158 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.654136 5801 generic.go:356] "Generic (PLEG): container finished" podID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerID="b7e7d3188cd510e938bf4835623a2034161095aa47f6b6e80f55a65db60776c4" exitCode=0 Oct 30 21:52:00.654281 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.654172 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" event={"ID":"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1","Type":"ContainerDied","Data":"b7e7d3188cd510e938bf4835623a2034161095aa47f6b6e80f55a65db60776c4"} Oct 30 21:52:00.654281 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.654187 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" Oct 30 21:52:00.654281 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.654194 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb" event={"ID":"bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1","Type":"ContainerDied","Data":"d76cd2f3ed15ecd4586e894a1b5013e8cd0f828d9f99392074c76c24f952ce7e"} Oct 30 21:52:00.663851 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.663832 5801 scope.go:122] "RemoveContainer" containerID="fe597c9e61aa17655ab23710562b507ead65ae66da9d138535e51044324e8375" Oct 30 21:52:00.664113 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:52:00.664095 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe597c9e61aa17655ab23710562b507ead65ae66da9d138535e51044324e8375\": container with ID starting with fe597c9e61aa17655ab23710562b507ead65ae66da9d138535e51044324e8375 not found: ID does not exist" containerID="fe597c9e61aa17655ab23710562b507ead65ae66da9d138535e51044324e8375" Oct 30 21:52:00.664172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.664120 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe597c9e61aa17655ab23710562b507ead65ae66da9d138535e51044324e8375"} err="failed to get container status \"fe597c9e61aa17655ab23710562b507ead65ae66da9d138535e51044324e8375\": rpc error: code = NotFound desc = could not find container \"fe597c9e61aa17655ab23710562b507ead65ae66da9d138535e51044324e8375\": container with ID starting with fe597c9e61aa17655ab23710562b507ead65ae66da9d138535e51044324e8375 not found: ID does not exist" Oct 30 21:52:00.664172 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.664140 5801 scope.go:122] "RemoveContainer" containerID="b7e7d3188cd510e938bf4835623a2034161095aa47f6b6e80f55a65db60776c4" Oct 30 21:52:00.672390 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.672374 5801 scope.go:122] "RemoveContainer" containerID="b7e7d3188cd510e938bf4835623a2034161095aa47f6b6e80f55a65db60776c4" Oct 30 21:52:00.672651 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:52:00.672633 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7e7d3188cd510e938bf4835623a2034161095aa47f6b6e80f55a65db60776c4\": container with ID starting with b7e7d3188cd510e938bf4835623a2034161095aa47f6b6e80f55a65db60776c4 not found: ID does not exist" containerID="b7e7d3188cd510e938bf4835623a2034161095aa47f6b6e80f55a65db60776c4" Oct 30 21:52:00.672722 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.672673 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7e7d3188cd510e938bf4835623a2034161095aa47f6b6e80f55a65db60776c4"} err="failed to get container status \"b7e7d3188cd510e938bf4835623a2034161095aa47f6b6e80f55a65db60776c4\": rpc error: code = NotFound desc = could not find container \"b7e7d3188cd510e938bf4835623a2034161095aa47f6b6e80f55a65db60776c4\": container with ID starting with b7e7d3188cd510e938bf4835623a2034161095aa47f6b6e80f55a65db60776c4 not found: ID does not exist" Oct 30 21:52:00.676262 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.676239 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p"] Oct 30 21:52:00.681281 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.681259 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-caa3dc/premium-w5-caa3dc-kserve-745c5bcb58-d274p"] Oct 30 21:52:00.693618 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.693597 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb"] Oct 30 21:52:00.698035 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.698010 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-caa3dc/simulator-w5-caa3dc-kserve-785c6bd7b7-vs9vb"] Oct 30 21:52:00.821147 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:00.821107 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:01.171646 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:01.171617 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:52:01.171839 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:01.171679 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" podUID="1fcdad2f-d069-41a7-ace4-49d0ba4abfaa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:52:01.523636 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:01.523594 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" path="/var/lib/kubelet/pods/bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1/volumes" Oct 30 21:52:01.524209 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:01.524187 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" path="/var/lib/kubelet/pods/d49a00ed-7af8-4999-91d7-ee8174080727/volumes" Oct 30 21:52:04.381916 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.381878 5801 patch_prober.go:29] interesting pod/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:52:04.382462 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.381938 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" podUID="e3976e0e-4921-46d0-99f4-2c26733ef15a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:52:04.404948 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.404905 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:04.516494 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.516457 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:04.851497 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.851474 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:52:04.885245 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.885219 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:52:04.961363 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.961292 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-token\" (UniqueName: \"kubernetes.io/projected/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-token\") pod \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " Oct 30 21:52:04.961363 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.961343 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-workload-certs\") pod \"e3976e0e-4921-46d0-99f4-2c26733ef15a\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " Oct 30 21:52:04.961626 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.961363 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-credential-socket\") pod \"e3976e0e-4921-46d0-99f4-2c26733ef15a\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " Oct 30 21:52:04.961626 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.961395 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3976e0e-4921-46d0-99f4-2c26733ef15a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3976e0e-4921-46d0-99f4-2c26733ef15a-istiod-ca-cert\") pod \"e3976e0e-4921-46d0-99f4-2c26733ef15a\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " Oct 30 21:52:04.961626 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.961424 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-credential-socket\") pod \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " Oct 30 21:52:04.961626 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.961453 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-kube-api-access-msv4z\" (UniqueName: \"kubernetes.io/projected/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-kube-api-access-msv4z\") pod \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " Oct 30 21:52:04.961626 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.961484 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-workload-socket\") pod \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " Oct 30 21:52:04.961626 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.961518 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-envoy\") pod \"e3976e0e-4921-46d0-99f4-2c26733ef15a\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " Oct 30 21:52:04.961626 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.961546 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-workload-socket\") pod \"e3976e0e-4921-46d0-99f4-2c26733ef15a\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " Oct 30 21:52:04.961626 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.961575 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-podinfo\") pod \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " Oct 30 21:52:04.961626 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.961607 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-workload-certs" pod "e3976e0e-4921-46d0-99f4-2c26733ef15a" (UID: "e3976e0e-4921-46d0-99f4-2c26733ef15a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:04.961626 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.961628 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-workload-certs\") pod \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " Oct 30 21:52:04.962210 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.961668 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-podinfo\") pod \"e3976e0e-4921-46d0-99f4-2c26733ef15a\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " Oct 30 21:52:04.962210 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.961714 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istiod-ca-cert\") pod \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " Oct 30 21:52:04.962210 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.961785 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3976e0e-4921-46d0-99f4-2c26733ef15a-kube-api-access-ft6jp\" (UniqueName: \"kubernetes.io/projected/e3976e0e-4921-46d0-99f4-2c26733ef15a-kube-api-access-ft6jp\") pod \"e3976e0e-4921-46d0-99f4-2c26733ef15a\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " Oct 30 21:52:04.962210 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.961826 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-envoy\") pod \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " Oct 30 21:52:04.962210 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.961865 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-data\") pod \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\" (UID: \"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa\") " Oct 30 21:52:04.962210 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.961897 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-data\") pod \"e3976e0e-4921-46d0-99f4-2c26733ef15a\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " Oct 30 21:52:04.962210 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.961928 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-token\" (UniqueName: \"kubernetes.io/projected/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-token\") pod \"e3976e0e-4921-46d0-99f4-2c26733ef15a\" (UID: \"e3976e0e-4921-46d0-99f4-2c26733ef15a\") " Oct 30 21:52:04.962613 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.962372 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:04.962873 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.961715 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-credential-socket" pod "e3976e0e-4921-46d0-99f4-2c26733ef15a" (UID: "e3976e0e-4921-46d0-99f4-2c26733ef15a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:04.962982 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.961840 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3976e0e-4921-46d0-99f4-2c26733ef15a-istiod-ca-cert" pod "e3976e0e-4921-46d0-99f4-2c26733ef15a" (UID: "e3976e0e-4921-46d0-99f4-2c26733ef15a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:04.962982 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.961959 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-workload-socket" pod "e3976e0e-4921-46d0-99f4-2c26733ef15a" (UID: "e3976e0e-4921-46d0-99f4-2c26733ef15a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:04.962982 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.962107 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-workload-certs" pod "1fcdad2f-d069-41a7-ace4-49d0ba4abfaa" (UID: "1fcdad2f-d069-41a7-ace4-49d0ba4abfaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:04.962982 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.962250 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istiod-ca-cert" pod "1fcdad2f-d069-41a7-ace4-49d0ba4abfaa" (UID: "1fcdad2f-d069-41a7-ace4-49d0ba4abfaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:04.962982 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.962533 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-credential-socket" pod "1fcdad2f-d069-41a7-ace4-49d0ba4abfaa" (UID: "1fcdad2f-d069-41a7-ace4-49d0ba4abfaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:04.962982 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.962705 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-workload-socket" pod "1fcdad2f-d069-41a7-ace4-49d0ba4abfaa" (UID: "1fcdad2f-d069-41a7-ace4-49d0ba4abfaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:04.962982 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.962850 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-data" pod "1fcdad2f-d069-41a7-ace4-49d0ba4abfaa" (UID: "1fcdad2f-d069-41a7-ace4-49d0ba4abfaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:04.963384 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.963089 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-data" pod "e3976e0e-4921-46d0-99f4-2c26733ef15a" (UID: "e3976e0e-4921-46d0-99f4-2c26733ef15a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:04.964209 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.964179 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-token" pod "1fcdad2f-d069-41a7-ace4-49d0ba4abfaa" (UID: "1fcdad2f-d069-41a7-ace4-49d0ba4abfaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:04.965465 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.965414 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-envoy" pod "e3976e0e-4921-46d0-99f4-2c26733ef15a" (UID: "e3976e0e-4921-46d0-99f4-2c26733ef15a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:04.965465 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.965424 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-token" pod "e3976e0e-4921-46d0-99f4-2c26733ef15a" (UID: "e3976e0e-4921-46d0-99f4-2c26733ef15a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:04.965631 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.965547 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-podinfo" pod "e3976e0e-4921-46d0-99f4-2c26733ef15a" (UID: "e3976e0e-4921-46d0-99f4-2c26733ef15a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:04.965631 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.965599 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-kube-api-access-msv4z" pod "1fcdad2f-d069-41a7-ace4-49d0ba4abfaa" (UID: "1fcdad2f-d069-41a7-ace4-49d0ba4abfaa"). InnerVolumeSpecName "kube-api-access-msv4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:04.965631 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.965601 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-podinfo" pod "1fcdad2f-d069-41a7-ace4-49d0ba4abfaa" (UID: "1fcdad2f-d069-41a7-ace4-49d0ba4abfaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:04.965789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.965681 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-envoy" pod "1fcdad2f-d069-41a7-ace4-49d0ba4abfaa" (UID: "1fcdad2f-d069-41a7-ace4-49d0ba4abfaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:04.965789 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:04.965691 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3976e0e-4921-46d0-99f4-2c26733ef15a-kube-api-access-ft6jp" pod "e3976e0e-4921-46d0-99f4-2c26733ef15a" (UID: "e3976e0e-4921-46d0-99f4-2c26733ef15a"). InnerVolumeSpecName "kube-api-access-ft6jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:05.063634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.063602 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:05.063634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.063627 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:05.063634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.063638 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:05.063888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.063646 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:05.063888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.063655 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:05.063888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.063665 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:05.063888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.063674 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3976e0e-4921-46d0-99f4-2c26733ef15a-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:05.063888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.063682 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:05.063888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.063692 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msv4z\" (UniqueName: \"kubernetes.io/projected/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-kube-api-access-msv4z\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:05.063888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.063702 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:05.063888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.063710 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:05.063888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.063718 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3976e0e-4921-46d0-99f4-2c26733ef15a-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:05.063888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.063726 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:05.063888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.063733 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:05.063888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.063741 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3976e0e-4921-46d0-99f4-2c26733ef15a-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:05.063888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.063774 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:05.063888 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.063783 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft6jp\" (UniqueName: \"kubernetes.io/projected/e3976e0e-4921-46d0-99f4-2c26733ef15a-kube-api-access-ft6jp\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:05.697279 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.697242 5801 generic.go:356] "Generic (PLEG): container finished" podID="e3976e0e-4921-46d0-99f4-2c26733ef15a" containerID="62dc366945d2c550b3e888b0e7d4b2ac39efa9bd3f4952e4310d1eb5ad2daa04" exitCode=0 Oct 30 21:52:05.697706 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.697304 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" Oct 30 21:52:05.697706 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.697327 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" event={"ID":"e3976e0e-4921-46d0-99f4-2c26733ef15a","Type":"ContainerDied","Data":"62dc366945d2c550b3e888b0e7d4b2ac39efa9bd3f4952e4310d1eb5ad2daa04"} Oct 30 21:52:05.697706 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.697417 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f" event={"ID":"e3976e0e-4921-46d0-99f4-2c26733ef15a","Type":"ContainerDied","Data":"9002f93bf2c66761355b5248bab3cf125c76063994ac84384851a88fce83ce76"} Oct 30 21:52:05.697706 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.697446 5801 scope.go:122] "RemoveContainer" containerID="62dc366945d2c550b3e888b0e7d4b2ac39efa9bd3f4952e4310d1eb5ad2daa04" Oct 30 21:52:05.698925 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.698906 5801 generic.go:356] "Generic (PLEG): container finished" podID="1fcdad2f-d069-41a7-ace4-49d0ba4abfaa" containerID="6bbadfdda2e87646a87cad137968f08f6b0e9b7972e31f1fc58d65c2eb5f43f0" exitCode=0 Oct 30 21:52:05.699042 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.698956 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" Oct 30 21:52:05.699042 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.698972 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" event={"ID":"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa","Type":"ContainerDied","Data":"6bbadfdda2e87646a87cad137968f08f6b0e9b7972e31f1fc58d65c2eb5f43f0"} Oct 30 21:52:05.699042 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.699013 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5" event={"ID":"1fcdad2f-d069-41a7-ace4-49d0ba4abfaa","Type":"ContainerDied","Data":"87e82958e8ed8dbe3834959ac713d3519ef243222b443fd9be267be2cd1b6175"} Oct 30 21:52:05.707807 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.707613 5801 scope.go:122] "RemoveContainer" containerID="62dc366945d2c550b3e888b0e7d4b2ac39efa9bd3f4952e4310d1eb5ad2daa04" Oct 30 21:52:05.707899 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:52:05.707879 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62dc366945d2c550b3e888b0e7d4b2ac39efa9bd3f4952e4310d1eb5ad2daa04\": container with ID starting with 62dc366945d2c550b3e888b0e7d4b2ac39efa9bd3f4952e4310d1eb5ad2daa04 not found: ID does not exist" containerID="62dc366945d2c550b3e888b0e7d4b2ac39efa9bd3f4952e4310d1eb5ad2daa04" Oct 30 21:52:05.707950 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.707906 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62dc366945d2c550b3e888b0e7d4b2ac39efa9bd3f4952e4310d1eb5ad2daa04"} err="failed to get container status \"62dc366945d2c550b3e888b0e7d4b2ac39efa9bd3f4952e4310d1eb5ad2daa04\": rpc error: code = NotFound desc = could not find container \"62dc366945d2c550b3e888b0e7d4b2ac39efa9bd3f4952e4310d1eb5ad2daa04\": container with ID starting with 62dc366945d2c550b3e888b0e7d4b2ac39efa9bd3f4952e4310d1eb5ad2daa04 not found: ID does not exist" Oct 30 21:52:05.707950 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.707922 5801 scope.go:122] "RemoveContainer" containerID="6bbadfdda2e87646a87cad137968f08f6b0e9b7972e31f1fc58d65c2eb5f43f0" Oct 30 21:52:05.719211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.717622 5801 scope.go:122] "RemoveContainer" containerID="6bbadfdda2e87646a87cad137968f08f6b0e9b7972e31f1fc58d65c2eb5f43f0" Oct 30 21:52:05.719211 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:52:05.717953 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bbadfdda2e87646a87cad137968f08f6b0e9b7972e31f1fc58d65c2eb5f43f0\": container with ID starting with 6bbadfdda2e87646a87cad137968f08f6b0e9b7972e31f1fc58d65c2eb5f43f0 not found: ID does not exist" containerID="6bbadfdda2e87646a87cad137968f08f6b0e9b7972e31f1fc58d65c2eb5f43f0" Oct 30 21:52:05.719211 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.717983 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bbadfdda2e87646a87cad137968f08f6b0e9b7972e31f1fc58d65c2eb5f43f0"} err="failed to get container status \"6bbadfdda2e87646a87cad137968f08f6b0e9b7972e31f1fc58d65c2eb5f43f0\": rpc error: code = NotFound desc = could not find container \"6bbadfdda2e87646a87cad137968f08f6b0e9b7972e31f1fc58d65c2eb5f43f0\": container with ID starting with 6bbadfdda2e87646a87cad137968f08f6b0e9b7972e31f1fc58d65c2eb5f43f0 not found: ID does not exist" Oct 30 21:52:05.725958 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.725936 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5"] Oct 30 21:52:05.728482 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.728459 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-6rjz5"] Oct 30 21:52:05.739653 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.739634 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f"] Oct 30 21:52:05.745396 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:05.745376 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d8034-openshift-default-74b55b95f9-tc65f"] Oct 30 21:52:07.007890 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:07.007850 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:07.503230 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:07.503194 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:07.526286 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:07.526255 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fcdad2f-d069-41a7-ace4-49d0ba4abfaa" path="/var/lib/kubelet/pods/1fcdad2f-d069-41a7-ace4-49d0ba4abfaa/volumes" Oct 30 21:52:07.526718 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:07.526704 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3976e0e-4921-46d0-99f4-2c26733ef15a" path="/var/lib/kubelet/pods/e3976e0e-4921-46d0-99f4-2c26733ef15a/volumes" Oct 30 21:52:07.715685 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:07.715644 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6"] Oct 30 21:52:07.720457 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:07.720425 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-caa3dc-t6fz6"] Oct 30 21:52:07.726286 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:07.724830 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:07.994736 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:07.994700 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:08.321778 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:08.321723 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:09.522929 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:09.522896 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa0e622c-88d2-49b6-a7ea-8617987ee929" path="/var/lib/kubelet/pods/fa0e622c-88d2-49b6-a7ea-8617987ee929/volumes" Oct 30 21:52:10.504026 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:10.503992 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:10.883313 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:10.883284 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:12.884503 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:12.884411 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:16.031651 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:16.031606 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl"] Oct 30 21:52:16.032184 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:16.031828 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" podUID="c1615c58-e876-4519-b38d-45166e1411c5" containerName="istio-proxy" containerID="cri-o://1fc16a853628620bf0d44a99bb9e2542a6e7e9430032942ad14e01f2e34c9da3" gracePeriod=30 Oct 30 21:52:16.038802 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:16.038742 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n"] Oct 30 21:52:16.039183 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:16.039155 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" podUID="f205247c-2c3c-4dc3-a3d2-b46066918ae3" containerName="istio-proxy" containerID="cri-o://a65c88cc36092db46b9a2e5e65de11a0cc90ba9c4bb08da88864c9d85da15eb7" gracePeriod=30 Oct 30 21:52:16.051816 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:16.051788 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01540444-bx7tl"] Oct 30 21:52:16.057001 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:16.056972 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01540444-bx7tl"] Oct 30 21:52:16.535696 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:16.535657 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:16.642713 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:16.642678 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:16.667821 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:16.667793 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:52:16.667944 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:16.667844 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" podUID="c1615c58-e876-4519-b38d-45166e1411c5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:52:17.310600 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:17.310559 5801 patch_prober.go:29] interesting pod/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:52:17.311026 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:17.310619 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" podUID="f205247c-2c3c-4dc3-a3d2-b46066918ae3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:52:17.522909 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:17.522877 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fb89951-0ce2-4023-a112-7ea5fd56b4a2" path="/var/lib/kubelet/pods/4fb89951-0ce2-4023-a112-7ea5fd56b4a2/volumes" Oct 30 21:52:21.363841 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.362470 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:21.403512 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.401852 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:52:21.420866 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.417022 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:52:21.479850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.478697 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1615c58-e876-4519-b38d-45166e1411c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1615c58-e876-4519-b38d-45166e1411c5-istiod-ca-cert\") pod \"c1615c58-e876-4519-b38d-45166e1411c5\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " Oct 30 21:52:21.479850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.478817 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-istio-data\") pod \"c1615c58-e876-4519-b38d-45166e1411c5\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " Oct 30 21:52:21.479850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.478856 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1615c58-e876-4519-b38d-45166e1411c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1615c58-e876-4519-b38d-45166e1411c5-istio-podinfo\") pod \"c1615c58-e876-4519-b38d-45166e1411c5\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " Oct 30 21:52:21.479850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.478894 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1615c58-e876-4519-b38d-45166e1411c5-kube-api-access-cdcj9\" (UniqueName: \"kubernetes.io/projected/c1615c58-e876-4519-b38d-45166e1411c5-kube-api-access-cdcj9\") pod \"c1615c58-e876-4519-b38d-45166e1411c5\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " Oct 30 21:52:21.479850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.478933 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1615c58-e876-4519-b38d-45166e1411c5-istio-token\" (UniqueName: \"kubernetes.io/projected/c1615c58-e876-4519-b38d-45166e1411c5-istio-token\") pod \"c1615c58-e876-4519-b38d-45166e1411c5\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " Oct 30 21:52:21.479850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.478989 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-istio-envoy\") pod \"c1615c58-e876-4519-b38d-45166e1411c5\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " Oct 30 21:52:21.479850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.479031 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-workload-certs\") pod \"c1615c58-e876-4519-b38d-45166e1411c5\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " Oct 30 21:52:21.479850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.479102 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-workload-socket\") pod \"c1615c58-e876-4519-b38d-45166e1411c5\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " Oct 30 21:52:21.479850 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.479174 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-credential-socket\") pod \"c1615c58-e876-4519-b38d-45166e1411c5\" (UID: \"c1615c58-e876-4519-b38d-45166e1411c5\") " Oct 30 21:52:21.480625 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.480598 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-credential-socket" pod "c1615c58-e876-4519-b38d-45166e1411c5" (UID: "c1615c58-e876-4519-b38d-45166e1411c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:21.481106 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.481082 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1615c58-e876-4519-b38d-45166e1411c5-istiod-ca-cert" pod "c1615c58-e876-4519-b38d-45166e1411c5" (UID: "c1615c58-e876-4519-b38d-45166e1411c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:21.481486 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.481466 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-istio-data" pod "c1615c58-e876-4519-b38d-45166e1411c5" (UID: "c1615c58-e876-4519-b38d-45166e1411c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:21.482876 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.482845 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-workload-socket" pod "c1615c58-e876-4519-b38d-45166e1411c5" (UID: "c1615c58-e876-4519-b38d-45166e1411c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:21.483171 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.483145 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-workload-certs" pod "c1615c58-e876-4519-b38d-45166e1411c5" (UID: "c1615c58-e876-4519-b38d-45166e1411c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:21.509834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.492091 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-istio-envoy" pod "c1615c58-e876-4519-b38d-45166e1411c5" (UID: "c1615c58-e876-4519-b38d-45166e1411c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:21.509834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.492177 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1615c58-e876-4519-b38d-45166e1411c5-istio-token" pod "c1615c58-e876-4519-b38d-45166e1411c5" (UID: "c1615c58-e876-4519-b38d-45166e1411c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:21.509834 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.492225 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1615c58-e876-4519-b38d-45166e1411c5-kube-api-access-cdcj9" pod "c1615c58-e876-4519-b38d-45166e1411c5" (UID: "c1615c58-e876-4519-b38d-45166e1411c5"). InnerVolumeSpecName "kube-api-access-cdcj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:21.617504 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.617401 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istiod-ca-cert\") pod \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " Oct 30 21:52:21.645775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.642859 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-workload-certs\") pod \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " Oct 30 21:52:21.645775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.642931 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1615c58-e876-4519-b38d-45166e1411c5-istio-podinfo" pod "c1615c58-e876-4519-b38d-45166e1411c5" (UID: "c1615c58-e876-4519-b38d-45166e1411c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:21.645775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.642983 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-envoy\") pod \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " Oct 30 21:52:21.645775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.643018 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-credential-socket\") pod \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " Oct 30 21:52:21.645775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.643058 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-workload-socket\") pod \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " Oct 30 21:52:21.645775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.643108 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-data\") pod \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " Oct 30 21:52:21.645775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.643143 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-token\" (UniqueName: \"kubernetes.io/projected/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-token\") pod \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " Oct 30 21:52:21.645775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.643213 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-podinfo\") pod \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " Oct 30 21:52:21.645775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.643250 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f205247c-2c3c-4dc3-a3d2-b46066918ae3-kube-api-access-nbh28\" (UniqueName: \"kubernetes.io/projected/f205247c-2c3c-4dc3-a3d2-b46066918ae3-kube-api-access-nbh28\") pod \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\" (UID: \"f205247c-2c3c-4dc3-a3d2-b46066918ae3\") " Oct 30 21:52:21.645775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.643869 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:21.645775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.643901 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1615c58-e876-4519-b38d-45166e1411c5-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:21.645775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.643916 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:21.645775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.643932 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1615c58-e876-4519-b38d-45166e1411c5-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:21.645775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.643947 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdcj9\" (UniqueName: \"kubernetes.io/projected/c1615c58-e876-4519-b38d-45166e1411c5-kube-api-access-cdcj9\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:21.645775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.643968 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1615c58-e876-4519-b38d-45166e1411c5-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:21.645775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.643983 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:21.645775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.643997 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:21.645775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.644013 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1615c58-e876-4519-b38d-45166e1411c5-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:21.687534 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.646375 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-workload-certs" pod "f205247c-2c3c-4dc3-a3d2-b46066918ae3" (UID: "f205247c-2c3c-4dc3-a3d2-b46066918ae3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:21.687534 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.646809 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f205247c-2c3c-4dc3-a3d2-b46066918ae3-kube-api-access-nbh28" pod "f205247c-2c3c-4dc3-a3d2-b46066918ae3" (UID: "f205247c-2c3c-4dc3-a3d2-b46066918ae3"). InnerVolumeSpecName "kube-api-access-nbh28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:21.687534 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.647061 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-data" pod "f205247c-2c3c-4dc3-a3d2-b46066918ae3" (UID: "f205247c-2c3c-4dc3-a3d2-b46066918ae3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:21.687534 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.647332 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-credential-socket" pod "f205247c-2c3c-4dc3-a3d2-b46066918ae3" (UID: "f205247c-2c3c-4dc3-a3d2-b46066918ae3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:21.687534 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.647588 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-workload-socket" pod "f205247c-2c3c-4dc3-a3d2-b46066918ae3" (UID: "f205247c-2c3c-4dc3-a3d2-b46066918ae3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:21.687534 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.662934 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istiod-ca-cert" pod "f205247c-2c3c-4dc3-a3d2-b46066918ae3" (UID: "f205247c-2c3c-4dc3-a3d2-b46066918ae3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:21.737466 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.735463 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:21.747693 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.745985 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:21.747693 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.746021 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbh28\" (UniqueName: \"kubernetes.io/projected/f205247c-2c3c-4dc3-a3d2-b46066918ae3-kube-api-access-nbh28\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:21.747693 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.746047 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:21.747693 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.746093 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:21.747693 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.746110 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:21.747693 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.746132 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:21.764913 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.763926 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-podinfo" pod "f205247c-2c3c-4dc3-a3d2-b46066918ae3" (UID: "f205247c-2c3c-4dc3-a3d2-b46066918ae3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:21.764913 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.764447 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-token" pod "f205247c-2c3c-4dc3-a3d2-b46066918ae3" (UID: "f205247c-2c3c-4dc3-a3d2-b46066918ae3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:21.877835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.877551 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:21.877835 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:21.877599 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:22.104681 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.104544 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:22.120123 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.119389 5801 generic.go:356] "Generic (PLEG): container finished" podID="f205247c-2c3c-4dc3-a3d2-b46066918ae3" containerID="a65c88cc36092db46b9a2e5e65de11a0cc90ba9c4bb08da88864c9d85da15eb7" exitCode=0 Oct 30 21:52:22.120123 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.119625 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" Oct 30 21:52:22.132815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.131612 5801 generic.go:356] "Generic (PLEG): container finished" podID="c1615c58-e876-4519-b38d-45166e1411c5" containerID="1fc16a853628620bf0d44a99bb9e2542a6e7e9430032942ad14e01f2e34c9da3" exitCode=0 Oct 30 21:52:22.132815 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.131871 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" Oct 30 21:52:22.151503 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.151434 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-envoy" pod "f205247c-2c3c-4dc3-a3d2-b46066918ae3" (UID: "f205247c-2c3c-4dc3-a3d2-b46066918ae3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:22.179520 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.154803 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" event={"ID":"f205247c-2c3c-4dc3-a3d2-b46066918ae3","Type":"ContainerDied","Data":"a65c88cc36092db46b9a2e5e65de11a0cc90ba9c4bb08da88864c9d85da15eb7"} Oct 30 21:52:22.179520 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.154882 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n" event={"ID":"f205247c-2c3c-4dc3-a3d2-b46066918ae3","Type":"ContainerDied","Data":"6f74504589a577027e94db6ecb17a046e0fc77cace1429df4b932b40b7b3326a"} Oct 30 21:52:22.179520 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.154917 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" event={"ID":"c1615c58-e876-4519-b38d-45166e1411c5","Type":"ContainerDied","Data":"1fc16a853628620bf0d44a99bb9e2542a6e7e9430032942ad14e01f2e34c9da3"} Oct 30 21:52:22.179520 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.154945 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl" event={"ID":"c1615c58-e876-4519-b38d-45166e1411c5","Type":"ContainerDied","Data":"8e408ddf07e93287f733fbfd00b27f0e433d318da7d09a7c9f7d464c7c591d46"} Oct 30 21:52:22.179520 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.154977 5801 scope.go:122] "RemoveContainer" containerID="a65c88cc36092db46b9a2e5e65de11a0cc90ba9c4bb08da88864c9d85da15eb7" Oct 30 21:52:22.203070 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.203034 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f205247c-2c3c-4dc3-a3d2-b46066918ae3-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:52:22.257246 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.257220 5801 scope.go:122] "RemoveContainer" containerID="3e4a2b5342a6f8be4d987bb6df54b325b28bfe3f06f18725669bc5941da34a73" Oct 30 21:52:22.352801 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.352542 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:22.465775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.464884 5801 scope.go:122] "RemoveContainer" containerID="a65c88cc36092db46b9a2e5e65de11a0cc90ba9c4bb08da88864c9d85da15eb7" Oct 30 21:52:22.472426 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:52:22.472139 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a65c88cc36092db46b9a2e5e65de11a0cc90ba9c4bb08da88864c9d85da15eb7\": container with ID starting with a65c88cc36092db46b9a2e5e65de11a0cc90ba9c4bb08da88864c9d85da15eb7 not found: ID does not exist" containerID="a65c88cc36092db46b9a2e5e65de11a0cc90ba9c4bb08da88864c9d85da15eb7" Oct 30 21:52:22.472426 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.472181 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a65c88cc36092db46b9a2e5e65de11a0cc90ba9c4bb08da88864c9d85da15eb7"} err="failed to get container status \"a65c88cc36092db46b9a2e5e65de11a0cc90ba9c4bb08da88864c9d85da15eb7\": rpc error: code = NotFound desc = could not find container \"a65c88cc36092db46b9a2e5e65de11a0cc90ba9c4bb08da88864c9d85da15eb7\": container with ID starting with a65c88cc36092db46b9a2e5e65de11a0cc90ba9c4bb08da88864c9d85da15eb7 not found: ID does not exist" Oct 30 21:52:22.472426 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.472204 5801 scope.go:122] "RemoveContainer" containerID="3e4a2b5342a6f8be4d987bb6df54b325b28bfe3f06f18725669bc5941da34a73" Oct 30 21:52:22.520803 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:52:22.500074 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4a2b5342a6f8be4d987bb6df54b325b28bfe3f06f18725669bc5941da34a73\": container with ID starting with 3e4a2b5342a6f8be4d987bb6df54b325b28bfe3f06f18725669bc5941da34a73 not found: ID does not exist" containerID="3e4a2b5342a6f8be4d987bb6df54b325b28bfe3f06f18725669bc5941da34a73" Oct 30 21:52:22.520803 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.500115 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4a2b5342a6f8be4d987bb6df54b325b28bfe3f06f18725669bc5941da34a73"} err="failed to get container status \"3e4a2b5342a6f8be4d987bb6df54b325b28bfe3f06f18725669bc5941da34a73\": rpc error: code = NotFound desc = could not find container \"3e4a2b5342a6f8be4d987bb6df54b325b28bfe3f06f18725669bc5941da34a73\": container with ID starting with 3e4a2b5342a6f8be4d987bb6df54b325b28bfe3f06f18725669bc5941da34a73 not found: ID does not exist" Oct 30 21:52:22.520803 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.500142 5801 scope.go:122] "RemoveContainer" containerID="1fc16a853628620bf0d44a99bb9e2542a6e7e9430032942ad14e01f2e34c9da3" Oct 30 21:52:22.521542 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.521497 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl"] Oct 30 21:52:22.542784 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.535605 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-zpqkl"] Oct 30 21:52:22.662875 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.662841 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n"] Oct 30 21:52:22.697826 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.680072 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-caa3dc-openshift-default-69ff984c7c-cgj5n"] Oct 30 21:52:22.777248 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.777221 5801 scope.go:122] "RemoveContainer" containerID="1fc16a853628620bf0d44a99bb9e2542a6e7e9430032942ad14e01f2e34c9da3" Oct 30 21:52:22.777955 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:52:22.777913 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fc16a853628620bf0d44a99bb9e2542a6e7e9430032942ad14e01f2e34c9da3\": container with ID starting with 1fc16a853628620bf0d44a99bb9e2542a6e7e9430032942ad14e01f2e34c9da3 not found: ID does not exist" containerID="1fc16a853628620bf0d44a99bb9e2542a6e7e9430032942ad14e01f2e34c9da3" Oct 30 21:52:22.778124 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:22.778091 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fc16a853628620bf0d44a99bb9e2542a6e7e9430032942ad14e01f2e34c9da3"} err="failed to get container status \"1fc16a853628620bf0d44a99bb9e2542a6e7e9430032942ad14e01f2e34c9da3\": rpc error: code = NotFound desc = could not find container \"1fc16a853628620bf0d44a99bb9e2542a6e7e9430032942ad14e01f2e34c9da3\": container with ID starting with 1fc16a853628620bf0d44a99bb9e2542a6e7e9430032942ad14e01f2e34c9da3 not found: ID does not exist" Oct 30 21:52:23.523767 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:23.523717 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1615c58-e876-4519-b38d-45166e1411c5" path="/var/lib/kubelet/pods/c1615c58-e876-4519-b38d-45166e1411c5/volumes" Oct 30 21:52:23.524375 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:23.524354 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f205247c-2c3c-4dc3-a3d2-b46066918ae3" path="/var/lib/kubelet/pods/f205247c-2c3c-4dc3-a3d2-b46066918ae3/volumes" Oct 30 21:52:23.767818 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:23.767783 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:29.903543 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:29.903509 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:30.277526 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:52:30.277445 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:52:52.471532 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:52.471490 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:53.035244 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:53.035206 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-833acf51-m7587"] Oct 30 21:52:53.040322 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:53.040292 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-833acf51-m7587"] Oct 30 21:52:53.524217 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:53.524173 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a471144f-13fa-42dc-870a-7022db45e7db" path="/var/lib/kubelet/pods/a471144f-13fa-42dc-870a-7022db45e7db/volumes" Oct 30 21:52:54.344309 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:54.344255 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:56.142151 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:56.142097 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:58.243859 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:52:58.243820 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:00.277105 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:53:00.277057 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:53:03.108702 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:53:03.108621 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:03.213871 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:53:03.213836 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:11.978541 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:53:11.978502 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:12.094662 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:53:12.094628 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:30.291691 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:53:30.291641 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:53:31.848775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:53:31.848208 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:53:31.848775 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:53:31.848311 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" podUID="493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:53:45.599983 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:53:45.599943 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:45.759017 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:53:45.758983 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:49.819473 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:53:49.819434 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-448hk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:53:49.820144 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:53:49.820092 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" podUID="79d7a2c8-0585-48a3-88d4-d49b23672216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:53:58.404503 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:53:58.404466 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:58.787011 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:53:58.786931 5801 scope.go:122] "RemoveContainer" containerID="b13b53269648edfc906c509f64cd08d2fce832b79c53f62c08e15aab522ed55d" Oct 30 21:53:58.796366 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:53:58.796346 5801 scope.go:122] "RemoveContainer" containerID="517c922dba9fc4d4bfe5a76dcadc236cc9d896c9726723b88f77e3035f2180a1" Oct 30 21:53:58.804572 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:53:58.804552 5801 scope.go:122] "RemoveContainer" containerID="97c9fde6445f086ab69966eb707ef92b6f6ddd225c14d41c40eef95273db0570" Oct 30 21:54:00.276957 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:54:00.276921 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:54:14.936394 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:14.936347 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:19.059870 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:19.059800 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63561e84-hqgzs"] Oct 30 21:54:19.083776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:19.080814 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63561e84-hqgzs"] Oct 30 21:54:19.085931 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:19.085902 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96ffa292-8b8gj"] Oct 30 21:54:19.123776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:19.103299 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96ffa292-8b8gj"] Oct 30 21:54:19.540566 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:19.540489 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d648b4b-c946-4b26-b77b-a3837e5b2135" path="/var/lib/kubelet/pods/0d648b4b-c946-4b26-b77b-a3837e5b2135/volumes" Oct 30 21:54:19.541496 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:19.541472 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eecf577b-fd05-4675-b050-5cf510babaa4" path="/var/lib/kubelet/pods/eecf577b-fd05-4675-b050-5cf510babaa4/volumes" Oct 30 21:54:19.811995 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:19.811952 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-448hk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:54:19.812312 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:19.812282 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" podUID="79d7a2c8-0585-48a3-88d4-d49b23672216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:21.646736 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:21.646695 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:30.277526 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:54:30.277481 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:54:30.998808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:30.998746 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:37.034607 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:37.034506 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d631ede9-h847l"] Oct 30 21:54:37.037616 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:37.037585 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d631ede9-h847l"] Oct 30 21:54:37.040629 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:37.040602 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc4da15-cqdfn"] Oct 30 21:54:37.043953 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:37.043930 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc4da15-cqdfn"] Oct 30 21:54:37.522189 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:37.522157 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8425a8c1-cff8-42cd-af91-c6b3b2f8d222" path="/var/lib/kubelet/pods/8425a8c1-cff8-42cd-af91-c6b3b2f8d222/volumes" Oct 30 21:54:37.522640 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:37.522625 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaace599-087f-4a86-91a6-2abd2a65cfaa" path="/var/lib/kubelet/pods/aaace599-087f-4a86-91a6-2abd2a65cfaa/volumes" Oct 30 21:54:51.890640 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:51.890602 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:52.022056 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:52.022007 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:58.428970 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:58.428931 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:58.833871 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:58.833835 5801 scope.go:122] "RemoveContainer" containerID="e4b55302b15a622bf0267e98cb85b3c2ec076083574a03753163048d08e5ee81" Oct 30 21:54:58.846193 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:58.846154 5801 scope.go:122] "RemoveContainer" containerID="667bbf2a2731b80db9bb3e3480fadf7a06c17cef395f31c259f0b3d83389c849" Oct 30 21:54:58.855469 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:58.855442 5801 scope.go:122] "RemoveContainer" containerID="2de5dc4f44715b56cdda960ee1f7dc791d742a91602124b61f3c65a78dd223b0" Oct 30 21:54:58.864358 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:58.864332 5801 scope.go:122] "RemoveContainer" containerID="7d83534b6450113b13484172576d7c633305e6ac9acd7ca6b2a8491a8e85f5b6" Oct 30 21:54:58.873192 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:54:58.873170 5801 scope.go:122] "RemoveContainer" containerID="ea3fa7099be1f01db676fcaa9bbf08af47a4968c2cce659adb171b7a2efbcd8d" Oct 30 21:55:00.276329 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:55:00.276290 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:55:01.696968 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:55:01.696931 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:13.586887 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:55:13.586843 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:16.841920 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:55:16.841885 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:55:16.842491 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:55:16.842422 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" podUID="493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:18.027969 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:55:18.027929 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58c5d82-cm72n"] Oct 30 21:55:18.031936 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:55:18.031911 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58c5d82-cm72n"] Oct 30 21:55:19.522985 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:55:19.522951 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce20b749-0d51-4e79-a243-109b758e70f8" path="/var/lib/kubelet/pods/ce20b749-0d51-4e79-a243-109b758e70f8/volumes" Oct 30 21:55:30.277061 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:55:30.277021 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:55:49.638804 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:55:49.638743 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:49.740853 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:55:49.740812 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:58.434680 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:55:58.434642 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:58.884870 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:55:58.884833 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:58.914721 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:55:58.914697 5801 scope.go:122] "RemoveContainer" containerID="d1c893f4837ddf0a02cabde274c4175fcaf15037eed25cd89b08f10722b32524" Oct 30 21:55:58.924255 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:55:58.924234 5801 scope.go:122] "RemoveContainer" containerID="a3b04c5a78be84f4b79b5245c510bbdbbf1891444537e4fb3f32b0192ffe9693" Oct 30 21:55:58.932621 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:55:58.932599 5801 scope.go:122] "RemoveContainer" containerID="f290e20df2e40cf53b11a519ac4cfcc746d6ab75d31f3cdf4778ea84a262829d" Oct 30 21:55:58.943166 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:55:58.943148 5801 scope.go:122] "RemoveContainer" containerID="e67dcfd46e67df02436914353a77a8d4d8a5c62dc688f7586d8972d374aa23a7" Oct 30 21:55:58.951049 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:55:58.951033 5801 scope.go:122] "RemoveContainer" containerID="63b7151335959dd223ebdf01c046ef23652805a21dae786f756d66e9c7df2361" Oct 30 21:55:58.958628 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:55:58.958604 5801 scope.go:122] "RemoveContainer" containerID="fdfa9918702e0234bc088474bdd102a7563ca0610d4695ea486ec4528c0598a2" Oct 30 21:55:58.966080 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:55:58.966065 5801 scope.go:122] "RemoveContainer" containerID="e12414833d773583cf64631985773fa63158e3c882d2043ef6ed8f584ba4ef04" Oct 30 21:55:58.990570 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:55:58.990553 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:00.277733 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:56:00.277693 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:56:22.033310 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:22.033081 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-760beae1-9k5tp"] Oct 30 21:56:22.038344 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:22.038313 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-760beae1-9k5tp"] Oct 30 21:56:23.536777 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:23.533664 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b" path="/var/lib/kubelet/pods/59a8b3a0-eecb-48a1-9ee3-d94ac73ae63b/volumes" Oct 30 21:56:29.025433 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:29.025399 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-473ccdb9-fsbfd"] Oct 30 21:56:29.033927 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:29.033898 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f664bea-rpgtd"] Oct 30 21:56:29.036358 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:29.036332 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-473ccdb9-fsbfd"] Oct 30 21:56:29.038037 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:29.038016 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f664bea-rpgtd"] Oct 30 21:56:29.522808 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:29.522776 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606" path="/var/lib/kubelet/pods/e4ae1bc2-03cd-4df3-bf3d-f04ad66e3606/volumes" Oct 30 21:56:29.523175 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:29.523161 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efb9944d-e392-451e-89a6-1d6030441138" path="/var/lib/kubelet/pods/efb9944d-e392-451e-89a6-1d6030441138/volumes" Oct 30 21:56:30.278816 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:56:30.278780 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:56:32.033642 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:32.033612 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9374298-5ln8q"] Oct 30 21:56:32.037002 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:32.036978 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9374298-5ln8q"] Oct 30 21:56:33.521745 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:33.521711 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44f975b9-767d-4a1c-87bd-e8043cdf84c0" path="/var/lib/kubelet/pods/44f975b9-767d-4a1c-87bd-e8043cdf84c0/volumes" Oct 30 21:56:37.026734 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:37.026697 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-212c9eb0-9fb99"] Oct 30 21:56:37.029479 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:37.029450 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3c67196-c2mnf"] Oct 30 21:56:37.031628 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:37.031604 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-212c9eb0-9fb99"] Oct 30 21:56:37.033019 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:37.032999 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3c67196-c2mnf"] Oct 30 21:56:37.522844 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:37.522809 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a8a85ae-4b64-473a-beac-cbde16b4ea4f" path="/var/lib/kubelet/pods/6a8a85ae-4b64-473a-beac-cbde16b4ea4f/volumes" Oct 30 21:56:37.523215 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:37.523202 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9e4efc8-551c-4352-a036-030ba1117dc3" path="/var/lib/kubelet/pods/b9e4efc8-551c-4352-a036-030ba1117dc3/volumes" Oct 30 21:56:49.821776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:49.820872 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-448hk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:56:49.821776 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:49.820982 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" podUID="79d7a2c8-0585-48a3-88d4-d49b23672216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:56:56.035208 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:56.035173 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8273be2-7k7tx"] Oct 30 21:56:56.042641 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:56.042614 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8273be2-7k7tx"] Oct 30 21:56:57.522215 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:57.522183 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01d7e580-7045-4135-beaf-03297571ed5a" path="/var/lib/kubelet/pods/01d7e580-7045-4135-beaf-03297571ed5a/volumes" Oct 30 21:56:58.496005 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:58.495967 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:59.010880 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:59.010844 5801 scope.go:122] "RemoveContainer" containerID="4c12a67a508d4cdb6c0fb0e5f38d50c3cbafd97adc877430570bee98f43b1a14" Oct 30 21:56:59.019972 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:59.019949 5801 scope.go:122] "RemoveContainer" containerID="8a7d8ee01470f3fc7d45bcb864f4f4b7e64771f1d52b26ebe4d8f7f511e7d8ee" Oct 30 21:56:59.028361 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:59.028338 5801 scope.go:122] "RemoveContainer" containerID="645609fad69b6467636bf55484e520127c99d109ea7d5b07fc0ef96e2d46ae29" Oct 30 21:56:59.036376 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:59.036352 5801 scope.go:122] "RemoveContainer" containerID="50865f66f989b811e6056c622767da272e9f76d5495fb27b0be3686708ac8d9f" Oct 30 21:56:59.044145 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:59.044126 5801 scope.go:122] "RemoveContainer" containerID="e5f0650bd0a87df55b1b9d31f4fdcb39ba38b15a1d20b9f97f862ed1f81b5e8b" Oct 30 21:56:59.051931 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:59.051912 5801 scope.go:122] "RemoveContainer" containerID="faaa1e75787b360f0c15c5880c38b9906a67fa44de248ef29b00b89707fd8d50" Oct 30 21:56:59.059317 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:59.059300 5801 scope.go:122] "RemoveContainer" containerID="2a349429c74c8f5062646a6a61c97a059f6a34358566863653562fb916797dfc" Oct 30 21:56:59.066956 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:59.066936 5801 scope.go:122] "RemoveContainer" containerID="3888199eff371d1575e24c5168dd6330f1bad78de23f04a6cf5fc74e9f6f91d8" Oct 30 21:56:59.074666 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:59.074647 5801 scope.go:122] "RemoveContainer" containerID="d646ec5ac332821664a6f420f4778cacfcf3f94d36d05b9154cef80d81ea128b" Oct 30 21:56:59.082780 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:56:59.082748 5801 scope.go:122] "RemoveContainer" containerID="ac1cdae13c15fa97882230b8062f5efbc74f6e9dc26b01a47719a4339de103ce" Oct 30 21:57:00.278286 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:57:00.278249 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:57:06.461148 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:06.461111 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:06.575685 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:06.575640 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:18.970316 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:18.970277 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769"] Oct 30 21:57:18.970740 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:18.970514 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" podUID="eb0e7692-fb76-4697-a947-a0ff3bfd72ba" containerName="discovery" containerID="cri-o://e6840060f3c8074f923f7c1cfc3baf04a8486d5ef002143fbf1561e4edc81b1d" gracePeriod=30 Oct 30 21:57:19.218726 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.218704 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:57:19.245038 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.244988 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-local-certs\") pod \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " Oct 30 21:57:19.245162 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.245045 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-kube-api-access-pksgp\" (UniqueName: \"kubernetes.io/projected/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-kube-api-access-pksgp\") pod \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " Oct 30 21:57:19.245162 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.245065 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-csr-dns-cert\") pod \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " Oct 30 21:57:19.245162 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.245095 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-csr-ca-configmap\") pod \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " Oct 30 21:57:19.245641 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.245267 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-token\" (UniqueName: \"kubernetes.io/projected/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-token\") pod \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " Oct 30 21:57:19.245641 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.245357 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-kubeconfig\") pod \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " Oct 30 21:57:19.245641 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.245414 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-cacerts\" (UniqueName: \"kubernetes.io/secret/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-cacerts\") pod \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\" (UID: \"eb0e7692-fb76-4697-a947-a0ff3bfd72ba\") " Oct 30 21:57:19.245641 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.245597 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-csr-ca-configmap" pod "eb0e7692-fb76-4697-a947-a0ff3bfd72ba" (UID: "eb0e7692-fb76-4697-a947-a0ff3bfd72ba"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:19.247516 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.245911 5801 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-csr-ca-configmap\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:19.248271 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.248226 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-kubeconfig" pod "eb0e7692-fb76-4697-a947-a0ff3bfd72ba" (UID: "eb0e7692-fb76-4697-a947-a0ff3bfd72ba"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:19.248719 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.248683 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-cacerts" pod "eb0e7692-fb76-4697-a947-a0ff3bfd72ba" (UID: "eb0e7692-fb76-4697-a947-a0ff3bfd72ba"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:19.249357 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.249336 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-local-certs" pod "eb0e7692-fb76-4697-a947-a0ff3bfd72ba" (UID: "eb0e7692-fb76-4697-a947-a0ff3bfd72ba"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:19.249477 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.249347 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-csr-dns-cert" pod "eb0e7692-fb76-4697-a947-a0ff3bfd72ba" (UID: "eb0e7692-fb76-4697-a947-a0ff3bfd72ba"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:19.250002 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.249975 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-kube-api-access-pksgp" pod "eb0e7692-fb76-4697-a947-a0ff3bfd72ba" (UID: "eb0e7692-fb76-4697-a947-a0ff3bfd72ba"). InnerVolumeSpecName "kube-api-access-pksgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:19.250398 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.250378 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-token" pod "eb0e7692-fb76-4697-a947-a0ff3bfd72ba" (UID: "eb0e7692-fb76-4697-a947-a0ff3bfd72ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:19.346547 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.346518 5801 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-cacerts\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:19.346547 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.346544 5801 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-local-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:19.346717 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.346555 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pksgp\" (UniqueName: \"kubernetes.io/projected/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-kube-api-access-pksgp\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:19.346717 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.346565 5801 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-csr-dns-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:19.346717 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.346574 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:19.346717 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.346582 5801 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eb0e7692-fb76-4697-a947-a0ff3bfd72ba-istio-kubeconfig\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:19.641617 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.641581 5801 generic.go:356] "Generic (PLEG): container finished" podID="eb0e7692-fb76-4697-a947-a0ff3bfd72ba" containerID="e6840060f3c8074f923f7c1cfc3baf04a8486d5ef002143fbf1561e4edc81b1d" exitCode=0 Oct 30 21:57:19.641791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.641624 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" Oct 30 21:57:19.641791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.641669 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" event={"ID":"eb0e7692-fb76-4697-a947-a0ff3bfd72ba","Type":"ContainerDied","Data":"e6840060f3c8074f923f7c1cfc3baf04a8486d5ef002143fbf1561e4edc81b1d"} Oct 30 21:57:19.641791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.641718 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769" event={"ID":"eb0e7692-fb76-4697-a947-a0ff3bfd72ba","Type":"ContainerDied","Data":"e3bbd8206991b594bde7b13abaf5e26cac63c1e44dea6c6bc9d9d49b47321bc3"} Oct 30 21:57:19.641791 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.641741 5801 scope.go:122] "RemoveContainer" containerID="e6840060f3c8074f923f7c1cfc3baf04a8486d5ef002143fbf1561e4edc81b1d" Oct 30 21:57:19.651149 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.651130 5801 scope.go:122] "RemoveContainer" containerID="e6840060f3c8074f923f7c1cfc3baf04a8486d5ef002143fbf1561e4edc81b1d" Oct 30 21:57:19.651400 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:57:19.651380 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6840060f3c8074f923f7c1cfc3baf04a8486d5ef002143fbf1561e4edc81b1d\": container with ID starting with e6840060f3c8074f923f7c1cfc3baf04a8486d5ef002143fbf1561e4edc81b1d not found: ID does not exist" containerID="e6840060f3c8074f923f7c1cfc3baf04a8486d5ef002143fbf1561e4edc81b1d" Oct 30 21:57:19.651472 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.651414 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6840060f3c8074f923f7c1cfc3baf04a8486d5ef002143fbf1561e4edc81b1d"} err="failed to get container status \"e6840060f3c8074f923f7c1cfc3baf04a8486d5ef002143fbf1561e4edc81b1d\": rpc error: code = NotFound desc = could not find container \"e6840060f3c8074f923f7c1cfc3baf04a8486d5ef002143fbf1561e4edc81b1d\": container with ID starting with e6840060f3c8074f923f7c1cfc3baf04a8486d5ef002143fbf1561e4edc81b1d not found: ID does not exist" Oct 30 21:57:19.659634 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.659612 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769"] Oct 30 21:57:19.663565 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:19.663533 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jv769"] Oct 30 21:57:21.524871 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:21.524829 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb0e7692-fb76-4697-a947-a0ff3bfd72ba" path="/var/lib/kubelet/pods/eb0e7692-fb76-4697-a947-a0ff3bfd72ba/volumes" Oct 30 21:57:30.277908 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:57:30.277868 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:57:35.716627 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:35.716524 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:36.598823 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:36.598789 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:37.994660 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:37.994626 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:38.107239 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:38.107204 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:53.751601 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:53.751564 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:53.850457 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:53.850408 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:58.540405 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:58.540349 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:59.142282 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:59.142249 5801 scope.go:122] "RemoveContainer" containerID="97e8f497bfad441622b58dc9273a9bb41c8afd0a92f167d11300a4c5ac47a348" Oct 30 21:57:59.151256 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:59.151231 5801 scope.go:122] "RemoveContainer" containerID="5ed01d4ddca6b34be16ee90b1865dd9d952a74d9badb564252f824d4770a4ca2" Oct 30 21:57:59.159283 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:59.159263 5801 scope.go:122] "RemoveContainer" containerID="1517550c5d3c80484620bbf73ecd70169c7210328d84c66729bdf39facb74269" Oct 30 21:57:59.166742 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:59.166724 5801 scope.go:122] "RemoveContainer" containerID="913f2c100dcbc127d995574ca82eac06c781b8207d54cf828487d27566759afe" Oct 30 21:57:59.174338 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:59.174320 5801 scope.go:122] "RemoveContainer" containerID="26e97e35207154a84f60d2c25740e764499296e737e774f1d0cbc4fae340f926" Oct 30 21:57:59.181606 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:59.181586 5801 scope.go:122] "RemoveContainer" containerID="2366b3748b4981a600ad55faf9c3238b7f8f6d966f368a6856aec800393a1266" Oct 30 21:57:59.188990 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:59.188971 5801 scope.go:122] "RemoveContainer" containerID="c36aa2da8f3a2296342436c6e00da945763e08711432479f089464787a076aac" Oct 30 21:57:59.196401 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:59.196383 5801 scope.go:122] "RemoveContainer" containerID="39ecc5d7f9f6bb1230e29c4e604d282222c9a4c48b1fc03391cabc2cbb1c4bce" Oct 30 21:57:59.203745 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:59.203728 5801 scope.go:122] "RemoveContainer" containerID="812eb8885d0d2f74739acda04038f7503cc4bf724a3f78123f52d3b7103087ae" Oct 30 21:57:59.211215 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:59.211198 5801 scope.go:122] "RemoveContainer" containerID="e2d93c13d490a6bb78c809cf7cc1f0b06828d16ca7f7862ab09113a7c340dde2" Oct 30 21:57:59.219684 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:59.219665 5801 scope.go:122] "RemoveContainer" containerID="9c7c960f9b06f5949df7098b1efa99f0248a8571dcf3af5b613d82546996d5df" Oct 30 21:57:59.227066 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:59.227048 5801 scope.go:122] "RemoveContainer" containerID="5c1960554ad5f9cffdd0b2b1fc609f317ebe581eab035dd1a2cd82d6b84b92fa" Oct 30 21:57:59.234244 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:57:59.234215 5801 scope.go:122] "RemoveContainer" containerID="caad361754218494e220b04a6e1a8dd7d684f86b593890f3abf426ef79890e59" Oct 30 21:58:00.277628 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:58:00.277588 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:58:30.278244 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:58:30.278204 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:58:58.601060 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:58:58.601026 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:00.277602 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:59:00.277567 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:59:06.910582 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:59:06.910505 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:30.278310 ip-10-0-140-163 kubenswrapper[5801]: E1030 21:59:30.278269 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 21:59:44.218903 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:59:44.218865 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:44.320967 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:59:44.320924 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:58.654412 ip-10-0-140-163 kubenswrapper[5801]: I1030 21:59:58.654372 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:00.152744 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.152709 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69"] Oct 30 22:00:00.153189 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153113 5801 cpu_manager.go:414] "containerMap: removing container" podUID="eb0e7692-fb76-4697-a947-a0ff3bfd72ba" containerName="discovery" Oct 30 22:00:00.153189 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153127 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb0e7692-fb76-4697-a947-a0ff3bfd72ba" containerName="discovery" Oct 30 22:00:00.153189 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153138 5801 cpu_manager.go:414] "containerMap: removing container" podUID="f205247c-2c3c-4dc3-a3d2-b46066918ae3" containerName="istio-proxy" Oct 30 22:00:00.153189 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153143 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f205247c-2c3c-4dc3-a3d2-b46066918ae3" containerName="istio-proxy" Oct 30 22:00:00.153189 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153150 5801 cpu_manager.go:414] "containerMap: removing container" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" Oct 30 22:00:00.153189 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153155 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" Oct 30 22:00:00.153189 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153162 5801 cpu_manager.go:414] "containerMap: removing container" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" Oct 30 22:00:00.153189 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153166 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" Oct 30 22:00:00.153189 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153174 5801 cpu_manager.go:414] "containerMap: removing container" podUID="01d7e580-7045-4135-beaf-03297571ed5a" containerName="revoke-keys" Oct 30 22:00:00.153189 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153178 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01d7e580-7045-4135-beaf-03297571ed5a" containerName="revoke-keys" Oct 30 22:00:00.153189 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153189 5801 cpu_manager.go:414] "containerMap: removing container" podUID="0f258bc9-d2fd-446a-9154-36caf951e081" containerName="istio-proxy" Oct 30 22:00:00.153189 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153194 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f258bc9-d2fd-446a-9154-36caf951e081" containerName="istio-proxy" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153206 5801 cpu_manager.go:414] "containerMap: removing container" podUID="e3976e0e-4921-46d0-99f4-2c26733ef15a" containerName="istio-proxy" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153211 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3976e0e-4921-46d0-99f4-2c26733ef15a" containerName="istio-proxy" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153218 5801 cpu_manager.go:414] "containerMap: removing container" podUID="f205247c-2c3c-4dc3-a3d2-b46066918ae3" containerName="istio-proxy" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153222 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f205247c-2c3c-4dc3-a3d2-b46066918ae3" containerName="istio-proxy" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153229 5801 cpu_manager.go:414] "containerMap: removing container" podUID="9a837073-7227-4ae9-9338-a70202213538" containerName="istio-proxy" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153234 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a837073-7227-4ae9-9338-a70202213538" containerName="istio-proxy" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153245 5801 cpu_manager.go:414] "containerMap: removing container" podUID="1fcdad2f-d069-41a7-ace4-49d0ba4abfaa" containerName="istio-proxy" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153250 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fcdad2f-d069-41a7-ace4-49d0ba4abfaa" containerName="istio-proxy" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153265 5801 cpu_manager.go:414] "containerMap: removing container" podUID="c1615c58-e876-4519-b38d-45166e1411c5" containerName="istio-proxy" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153271 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1615c58-e876-4519-b38d-45166e1411c5" containerName="istio-proxy" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153283 5801 cpu_manager.go:414] "containerMap: removing container" podUID="ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" containerName="istio-proxy" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153287 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" containerName="istio-proxy" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153293 5801 cpu_manager.go:414] "containerMap: removing container" podUID="ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" containerName="istio-proxy" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153297 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" containerName="istio-proxy" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153351 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f258bc9-d2fd-446a-9154-36caf951e081" containerName="istio-proxy" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153360 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" containerName="istio-proxy" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153367 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb0e7692-fb76-4697-a947-a0ff3bfd72ba" containerName="discovery" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153373 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="f205247c-2c3c-4dc3-a3d2-b46066918ae3" containerName="istio-proxy" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153378 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="d49a00ed-7af8-4999-91d7-ee8174080727" containerName="main" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153385 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="bde5f9ed-f2d9-43ca-b3a9-8389dbb93ed1" containerName="main" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153393 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="01d7e580-7045-4135-beaf-03297571ed5a" containerName="revoke-keys" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153400 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f258bc9-d2fd-446a-9154-36caf951e081" containerName="istio-proxy" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153410 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3976e0e-4921-46d0-99f4-2c26733ef15a" containerName="istio-proxy" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153419 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="f205247c-2c3c-4dc3-a3d2-b46066918ae3" containerName="istio-proxy" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153425 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a837073-7227-4ae9-9338-a70202213538" containerName="istio-proxy" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153436 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fcdad2f-d069-41a7-ace4-49d0ba4abfaa" containerName="istio-proxy" Oct 30 22:00:00.153646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.153447 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1615c58-e876-4519-b38d-45166e1411c5" containerName="istio-proxy" Oct 30 22:00:00.156711 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.156692 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69" Oct 30 22:00:00.160614 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.160593 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 22:00:00.161735 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.161714 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:00:00.161735 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.161720 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:00:00.178128 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.178101 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69"] Oct 30 22:00:00.263345 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.263302 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrwmv\" (UniqueName: \"kubernetes.io/projected/f82d707e-48b5-4272-ad07-d9adf2296d97-kube-api-access-qrwmv\") pod \"maas-api-key-cleanup-29889960-7vd69\" (UID: \"f82d707e-48b5-4272-ad07-d9adf2296d97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69" Oct 30 22:00:00.263513 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.263411 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f82d707e-48b5-4272-ad07-d9adf2296d97-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-7vd69\" (UID: \"f82d707e-48b5-4272-ad07-d9adf2296d97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69" Oct 30 22:00:00.277522 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:00:00.277488 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:00:00.364496 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.364442 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrwmv\" (UniqueName: \"kubernetes.io/projected/f82d707e-48b5-4272-ad07-d9adf2296d97-kube-api-access-qrwmv\") pod \"maas-api-key-cleanup-29889960-7vd69\" (UID: \"f82d707e-48b5-4272-ad07-d9adf2296d97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69" Oct 30 22:00:00.364496 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.364503 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f82d707e-48b5-4272-ad07-d9adf2296d97-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-7vd69\" (UID: \"f82d707e-48b5-4272-ad07-d9adf2296d97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69" Oct 30 22:00:00.365141 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.365119 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f82d707e-48b5-4272-ad07-d9adf2296d97-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-7vd69\" (UID: \"f82d707e-48b5-4272-ad07-d9adf2296d97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69" Oct 30 22:00:00.377073 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.377045 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrwmv\" (UniqueName: \"kubernetes.io/projected/f82d707e-48b5-4272-ad07-d9adf2296d97-kube-api-access-qrwmv\") pod \"maas-api-key-cleanup-29889960-7vd69\" (UID: \"f82d707e-48b5-4272-ad07-d9adf2296d97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69" Oct 30 22:00:00.469874 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.469793 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69" Oct 30 22:00:00.744845 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:00.744819 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69"] Oct 30 22:00:00.747057 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:00:00.747033 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf82d707e_48b5_4272_ad07_d9adf2296d97.slice/crio-2d38c1af24211e901162553460b7e31f5b5bacc0286b75d1eb66460a8d3ad590 WatchSource:0}: Error finding container 2d38c1af24211e901162553460b7e31f5b5bacc0286b75d1eb66460a8d3ad590: Status 404 returned error can't find the container with id 2d38c1af24211e901162553460b7e31f5b5bacc0286b75d1eb66460a8d3ad590 Oct 30 22:00:01.435339 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:01.435249 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69" event={"ID":"f82d707e-48b5-4272-ad07-d9adf2296d97","Type":"ContainerStarted","Data":"7f2e121a9d5380c578b47c647464c4a9d062457dae4e7920f0753e69dc5c773f"} Oct 30 22:00:01.435339 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:01.435335 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69" event={"ID":"f82d707e-48b5-4272-ad07-d9adf2296d97","Type":"ContainerStarted","Data":"2d38c1af24211e901162553460b7e31f5b5bacc0286b75d1eb66460a8d3ad590"} Oct 30 22:00:01.455794 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:01.455721 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69" podStartSLOduration=1.455704922 podStartE2EDuration="1.455704922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.454013742 +0000 UTC m=+1870.049830422" watchObservedRunningTime="2026-10-30 22:00:01.455704922 +0000 UTC m=+1870.051521583" Oct 30 22:00:02.442619 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:02.442586 5801 generic.go:356] "Generic (PLEG): container finished" podID="f82d707e-48b5-4272-ad07-d9adf2296d97" containerID="7f2e121a9d5380c578b47c647464c4a9d062457dae4e7920f0753e69dc5c773f" exitCode=60 Oct 30 22:00:02.443084 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:02.442657 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69" event={"ID":"f82d707e-48b5-4272-ad07-d9adf2296d97","Type":"ContainerDied","Data":"7f2e121a9d5380c578b47c647464c4a9d062457dae4e7920f0753e69dc5c773f"} Oct 30 22:00:02.443084 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:02.442908 5801 scope.go:122] "RemoveContainer" containerID="7f2e121a9d5380c578b47c647464c4a9d062457dae4e7920f0753e69dc5c773f" Oct 30 22:00:03.451219 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:03.451132 5801 generic.go:356] "Generic (PLEG): container finished" podID="f82d707e-48b5-4272-ad07-d9adf2296d97" containerID="8faa2ad7e18fef04dab898965118760bdc588afe2073497553acf670323c377b" exitCode=60 Oct 30 22:00:03.451219 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:03.451207 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69" event={"ID":"f82d707e-48b5-4272-ad07-d9adf2296d97","Type":"ContainerDied","Data":"8faa2ad7e18fef04dab898965118760bdc588afe2073497553acf670323c377b"} Oct 30 22:00:03.451666 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:03.451263 5801 scope.go:122] "RemoveContainer" containerID="7f2e121a9d5380c578b47c647464c4a9d062457dae4e7920f0753e69dc5c773f" Oct 30 22:00:03.451666 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:03.451500 5801 scope.go:122] "RemoveContainer" containerID="8faa2ad7e18fef04dab898965118760bdc588afe2073497553acf670323c377b" Oct 30 22:00:03.451785 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:00:03.451720 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-7vd69_odh-ai-gateway-infra(f82d707e-48b5-4272-ad07-d9adf2296d97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69" podUID="f82d707e-48b5-4272-ad07-d9adf2296d97" Oct 30 22:00:04.458576 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:04.458544 5801 scope.go:122] "RemoveContainer" containerID="8faa2ad7e18fef04dab898965118760bdc588afe2073497553acf670323c377b" Oct 30 22:00:04.459020 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:00:04.458830 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-7vd69_odh-ai-gateway-infra(f82d707e-48b5-4272-ad07-d9adf2296d97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69" podUID="f82d707e-48b5-4272-ad07-d9adf2296d97" Oct 30 22:00:17.517804 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:17.517773 5801 scope.go:122] "RemoveContainer" containerID="8faa2ad7e18fef04dab898965118760bdc588afe2073497553acf670323c377b" Oct 30 22:00:18.529522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:18.529431 5801 generic.go:356] "Generic (PLEG): container finished" podID="f82d707e-48b5-4272-ad07-d9adf2296d97" containerID="1581d75dcff27c2cc10781b06f083c48f56d521c719e7cfad331db605a6b4213" exitCode=60 Oct 30 22:00:18.529522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:18.529500 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69" event={"ID":"f82d707e-48b5-4272-ad07-d9adf2296d97","Type":"ContainerDied","Data":"1581d75dcff27c2cc10781b06f083c48f56d521c719e7cfad331db605a6b4213"} Oct 30 22:00:18.530102 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:18.529559 5801 scope.go:122] "RemoveContainer" containerID="8faa2ad7e18fef04dab898965118760bdc588afe2073497553acf670323c377b" Oct 30 22:00:18.530102 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:18.529988 5801 scope.go:122] "RemoveContainer" containerID="1581d75dcff27c2cc10781b06f083c48f56d521c719e7cfad331db605a6b4213" Oct 30 22:00:18.530275 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:00:18.530253 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-7vd69_odh-ai-gateway-infra(f82d707e-48b5-4272-ad07-d9adf2296d97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69" podUID="f82d707e-48b5-4272-ad07-d9adf2296d97" Oct 30 22:00:19.558004 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:19.557965 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69"] Oct 30 22:00:19.696939 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:19.696908 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69" Oct 30 22:00:19.756890 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:19.756856 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f82d707e-48b5-4272-ad07-d9adf2296d97-kube-api-access-qrwmv\" (UniqueName: \"kubernetes.io/projected/f82d707e-48b5-4272-ad07-d9adf2296d97-kube-api-access-qrwmv\") pod \"f82d707e-48b5-4272-ad07-d9adf2296d97\" (UID: \"f82d707e-48b5-4272-ad07-d9adf2296d97\") " Oct 30 22:00:19.757051 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:19.756953 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f82d707e-48b5-4272-ad07-d9adf2296d97-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f82d707e-48b5-4272-ad07-d9adf2296d97-maas-api-service-ca\") pod \"f82d707e-48b5-4272-ad07-d9adf2296d97\" (UID: \"f82d707e-48b5-4272-ad07-d9adf2296d97\") " Oct 30 22:00:19.757411 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:19.757382 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f82d707e-48b5-4272-ad07-d9adf2296d97-maas-api-service-ca" pod "f82d707e-48b5-4272-ad07-d9adf2296d97" (UID: "f82d707e-48b5-4272-ad07-d9adf2296d97"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:19.758932 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:19.758902 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82d707e-48b5-4272-ad07-d9adf2296d97-kube-api-access-qrwmv" pod "f82d707e-48b5-4272-ad07-d9adf2296d97" (UID: "f82d707e-48b5-4272-ad07-d9adf2296d97"). InnerVolumeSpecName "kube-api-access-qrwmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:19.858352 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:19.858263 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrwmv\" (UniqueName: \"kubernetes.io/projected/f82d707e-48b5-4272-ad07-d9adf2296d97-kube-api-access-qrwmv\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:19.858352 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:19.858306 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f82d707e-48b5-4272-ad07-d9adf2296d97-maas-api-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:20.541443 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:20.541408 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69" Oct 30 22:00:20.541443 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:20.541420 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69" event={"ID":"f82d707e-48b5-4272-ad07-d9adf2296d97","Type":"ContainerDied","Data":"2d38c1af24211e901162553460b7e31f5b5bacc0286b75d1eb66460a8d3ad590"} Oct 30 22:00:20.541710 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:20.541476 5801 scope.go:122] "RemoveContainer" containerID="1581d75dcff27c2cc10781b06f083c48f56d521c719e7cfad331db605a6b4213" Oct 30 22:00:20.574825 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:20.574788 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69"] Oct 30 22:00:20.587431 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:20.587399 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-7vd69"] Oct 30 22:00:21.522539 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:21.522496 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f82d707e-48b5-4272-ad07-d9adf2296d97" path="/var/lib/kubelet/pods/f82d707e-48b5-4272-ad07-d9adf2296d97/volumes" Oct 30 22:00:30.279079 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:00:30.279042 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:00:58.715435 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:00:58.715398 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:00.276769 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:01:00.276710 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:01:30.276890 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:01:30.276847 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:01:58.767285 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:01:58.767250 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:00.277353 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:02:00.277311 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:02:21.270473 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.270387 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-krc6t"] Oct 30 22:02:21.270964 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.270775 5801 cpu_manager.go:414] "containerMap: removing container" podUID="f82d707e-48b5-4272-ad07-d9adf2296d97" containerName="cleanup" Oct 30 22:02:21.270964 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.270788 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f82d707e-48b5-4272-ad07-d9adf2296d97" containerName="cleanup" Oct 30 22:02:21.270964 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.270804 5801 cpu_manager.go:414] "containerMap: removing container" podUID="f82d707e-48b5-4272-ad07-d9adf2296d97" containerName="cleanup" Oct 30 22:02:21.270964 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.270810 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f82d707e-48b5-4272-ad07-d9adf2296d97" containerName="cleanup" Oct 30 22:02:21.270964 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.270831 5801 cpu_manager.go:414] "containerMap: removing container" podUID="0f258bc9-d2fd-446a-9154-36caf951e081" containerName="istio-proxy" Oct 30 22:02:21.270964 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.270837 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f258bc9-d2fd-446a-9154-36caf951e081" containerName="istio-proxy" Oct 30 22:02:21.270964 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.270896 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="f82d707e-48b5-4272-ad07-d9adf2296d97" containerName="cleanup" Oct 30 22:02:21.270964 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.270907 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="f82d707e-48b5-4272-ad07-d9adf2296d97" containerName="cleanup" Oct 30 22:02:21.270964 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.270916 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec34f7ef-9765-4b17-bdfe-cefd1f2a37b2" containerName="istio-proxy" Oct 30 22:02:21.274031 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.274014 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-krc6t" Oct 30 22:02:21.279277 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.279251 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:02:21.279513 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.279495 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 22:02:21.279782 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.279763 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:02:21.279917 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.279902 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:02:21.299500 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.299471 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-krc6t"] Oct 30 22:02:21.349845 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.349813 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqfbs\" (UniqueName: \"kubernetes.io/projected/1bdbaa88-f859-4e2a-8cee-8326a6b2fba9-kube-api-access-wqfbs\") pod \"maas-api-revoke-sub-3d4156f1-krc6t\" (UID: \"1bdbaa88-f859-4e2a-8cee-8326a6b2fba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-krc6t" Oct 30 22:02:21.350029 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.349856 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bdbaa88-f859-4e2a-8cee-8326a6b2fba9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-krc6t\" (UID: \"1bdbaa88-f859-4e2a-8cee-8326a6b2fba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-krc6t" Oct 30 22:02:21.350029 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.349881 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bdbaa88-f859-4e2a-8cee-8326a6b2fba9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-krc6t\" (UID: \"1bdbaa88-f859-4e2a-8cee-8326a6b2fba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-krc6t" Oct 30 22:02:21.451167 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.451136 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bdbaa88-f859-4e2a-8cee-8326a6b2fba9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-krc6t\" (UID: \"1bdbaa88-f859-4e2a-8cee-8326a6b2fba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-krc6t" Oct 30 22:02:21.451358 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.451236 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqfbs\" (UniqueName: \"kubernetes.io/projected/1bdbaa88-f859-4e2a-8cee-8326a6b2fba9-kube-api-access-wqfbs\") pod \"maas-api-revoke-sub-3d4156f1-krc6t\" (UID: \"1bdbaa88-f859-4e2a-8cee-8326a6b2fba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-krc6t" Oct 30 22:02:21.451358 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.451266 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bdbaa88-f859-4e2a-8cee-8326a6b2fba9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-krc6t\" (UID: \"1bdbaa88-f859-4e2a-8cee-8326a6b2fba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-krc6t" Oct 30 22:02:21.451905 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.451874 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bdbaa88-f859-4e2a-8cee-8326a6b2fba9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-krc6t\" (UID: \"1bdbaa88-f859-4e2a-8cee-8326a6b2fba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-krc6t" Oct 30 22:02:21.453585 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.453564 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bdbaa88-f859-4e2a-8cee-8326a6b2fba9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-krc6t\" (UID: \"1bdbaa88-f859-4e2a-8cee-8326a6b2fba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-krc6t" Oct 30 22:02:21.464227 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.464204 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqfbs\" (UniqueName: \"kubernetes.io/projected/1bdbaa88-f859-4e2a-8cee-8326a6b2fba9-kube-api-access-wqfbs\") pod \"maas-api-revoke-sub-3d4156f1-krc6t\" (UID: \"1bdbaa88-f859-4e2a-8cee-8326a6b2fba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-krc6t" Oct 30 22:02:21.586771 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.586734 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-krc6t" Oct 30 22:02:21.863996 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:21.863967 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-krc6t"] Oct 30 22:02:21.866525 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:02:21.866483 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bdbaa88_f859_4e2a_8cee_8326a6b2fba9.slice/crio-25be7b3f00ec0ecb07887531faf6277bf5d3d596e77cfe733602207fcb5f87e0 WatchSource:0}: Error finding container 25be7b3f00ec0ecb07887531faf6277bf5d3d596e77cfe733602207fcb5f87e0: Status 404 returned error can't find the container with id 25be7b3f00ec0ecb07887531faf6277bf5d3d596e77cfe733602207fcb5f87e0 Oct 30 22:02:22.135862 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:22.135771 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-krc6t" event={"ID":"1bdbaa88-f859-4e2a-8cee-8326a6b2fba9","Type":"ContainerStarted","Data":"25be7b3f00ec0ecb07887531faf6277bf5d3d596e77cfe733602207fcb5f87e0"} Oct 30 22:02:23.142807 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:23.142766 5801 generic.go:356] "Generic (PLEG): container finished" podID="1bdbaa88-f859-4e2a-8cee-8326a6b2fba9" containerID="6317d618ff4defba2f2e6e7a023bf8b066cee8ccd5d84ab1793d8d6ef00f97bf" exitCode=0 Oct 30 22:02:23.143217 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:23.142814 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-krc6t" event={"ID":"1bdbaa88-f859-4e2a-8cee-8326a6b2fba9","Type":"ContainerDied","Data":"6317d618ff4defba2f2e6e7a023bf8b066cee8ccd5d84ab1793d8d6ef00f97bf"} Oct 30 22:02:24.279911 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:24.279887 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-krc6t" Oct 30 22:02:24.381517 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:24.381486 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bdbaa88-f859-4e2a-8cee-8326a6b2fba9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bdbaa88-f859-4e2a-8cee-8326a6b2fba9-maas-api-xks-ca\") pod \"1bdbaa88-f859-4e2a-8cee-8326a6b2fba9\" (UID: \"1bdbaa88-f859-4e2a-8cee-8326a6b2fba9\") " Oct 30 22:02:24.381694 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:24.381574 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bdbaa88-f859-4e2a-8cee-8326a6b2fba9-kube-api-access-wqfbs\" (UniqueName: \"kubernetes.io/projected/1bdbaa88-f859-4e2a-8cee-8326a6b2fba9-kube-api-access-wqfbs\") pod \"1bdbaa88-f859-4e2a-8cee-8326a6b2fba9\" (UID: \"1bdbaa88-f859-4e2a-8cee-8326a6b2fba9\") " Oct 30 22:02:24.381694 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:24.381637 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bdbaa88-f859-4e2a-8cee-8326a6b2fba9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bdbaa88-f859-4e2a-8cee-8326a6b2fba9-maas-api-openshift-ca\") pod \"1bdbaa88-f859-4e2a-8cee-8326a6b2fba9\" (UID: \"1bdbaa88-f859-4e2a-8cee-8326a6b2fba9\") " Oct 30 22:02:24.382097 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:24.382066 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bdbaa88-f859-4e2a-8cee-8326a6b2fba9-maas-api-openshift-ca" pod "1bdbaa88-f859-4e2a-8cee-8326a6b2fba9" (UID: "1bdbaa88-f859-4e2a-8cee-8326a6b2fba9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:24.383596 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:24.383579 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bdbaa88-f859-4e2a-8cee-8326a6b2fba9-kube-api-access-wqfbs" pod "1bdbaa88-f859-4e2a-8cee-8326a6b2fba9" (UID: "1bdbaa88-f859-4e2a-8cee-8326a6b2fba9"). InnerVolumeSpecName "kube-api-access-wqfbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:24.383596 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:24.383582 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bdbaa88-f859-4e2a-8cee-8326a6b2fba9-maas-api-xks-ca" pod "1bdbaa88-f859-4e2a-8cee-8326a6b2fba9" (UID: "1bdbaa88-f859-4e2a-8cee-8326a6b2fba9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:24.482983 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:24.482898 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bdbaa88-f859-4e2a-8cee-8326a6b2fba9-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:24.482983 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:24.482930 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqfbs\" (UniqueName: \"kubernetes.io/projected/1bdbaa88-f859-4e2a-8cee-8326a6b2fba9-kube-api-access-wqfbs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:24.482983 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:24.482944 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bdbaa88-f859-4e2a-8cee-8326a6b2fba9-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.153770 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:25.153728 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-krc6t" Oct 30 22:02:25.153770 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:25.153766 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-krc6t" event={"ID":"1bdbaa88-f859-4e2a-8cee-8326a6b2fba9","Type":"ContainerDied","Data":"25be7b3f00ec0ecb07887531faf6277bf5d3d596e77cfe733602207fcb5f87e0"} Oct 30 22:02:25.154046 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:25.153807 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25be7b3f00ec0ecb07887531faf6277bf5d3d596e77cfe733602207fcb5f87e0" Oct 30 22:02:30.277585 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:02:30.277544 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:02:39.610128 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:39.610086 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:39.666238 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:39.666200 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm"] Oct 30 22:02:39.666539 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:39.666496 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" podUID="9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9" containerName="istio-proxy" containerID="cri-o://491aa761acede556be776655a81f800a3967ef7b917a11436202c5d2243e9e99" gracePeriod=30 Oct 30 22:02:39.668120 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:39.668092 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk"] Oct 30 22:02:39.668328 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:39.668303 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" podUID="79d7a2c8-0585-48a3-88d4-d49b23672216" containerName="istio-proxy" containerID="cri-o://8eeff9e295780e2ce15ac0f0d42f372d5844b94d28b2a1d5bdc026100dad48d5" gracePeriod=30 Oct 30 22:02:39.669686 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:39.669660 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr"] Oct 30 22:02:39.669898 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:39.669874 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" podUID="493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b" containerName="istio-proxy" containerID="cri-o://7556d3f99a0c67aa563c1911fa1b53ebbb0775b1569160dd71a41f9c8b83bcca" gracePeriod=30 Oct 30 22:02:40.142026 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.141989 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d"] Oct 30 22:02:40.142377 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.142365 5801 cpu_manager.go:414] "containerMap: removing container" podUID="1bdbaa88-f859-4e2a-8cee-8326a6b2fba9" containerName="revoke-keys" Oct 30 22:02:40.142430 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.142378 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bdbaa88-f859-4e2a-8cee-8326a6b2fba9" containerName="revoke-keys" Oct 30 22:02:40.142430 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.142390 5801 cpu_manager.go:414] "containerMap: removing container" podUID="f82d707e-48b5-4272-ad07-d9adf2296d97" containerName="cleanup" Oct 30 22:02:40.142430 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.142395 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f82d707e-48b5-4272-ad07-d9adf2296d97" containerName="cleanup" Oct 30 22:02:40.142543 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.142463 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="f82d707e-48b5-4272-ad07-d9adf2296d97" containerName="cleanup" Oct 30 22:02:40.142543 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.142473 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bdbaa88-f859-4e2a-8cee-8326a6b2fba9" containerName="revoke-keys" Oct 30 22:02:40.145861 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.145841 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.153944 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.153918 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-fa15d4-openshift-default-dockercfg-7j7q2\"" Oct 30 22:02:40.178340 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.176331 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d"] Oct 30 22:02:40.179103 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.179065 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-workload-certs\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.179356 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.179326 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-podinfo\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.179488 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.179386 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-workload-socket\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.179488 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.179437 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-data\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.179488 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.179469 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj5dd\" (UniqueName: \"kubernetes.io/projected/258e5cf7-89af-4f93-8423-24e4897e66e6-kube-api-access-cj5dd\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.179677 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.179502 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-envoy\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.179677 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.179540 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-credential-socket\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.179677 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.179579 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-token\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.179677 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.179634 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/258e5cf7-89af-4f93-8423-24e4897e66e6-istiod-ca-cert\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.280336 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.280289 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-data\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.280336 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.280332 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj5dd\" (UniqueName: \"kubernetes.io/projected/258e5cf7-89af-4f93-8423-24e4897e66e6-kube-api-access-cj5dd\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.280623 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.280366 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-envoy\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.280623 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.280400 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-credential-socket\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.280623 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.280433 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-token\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.280623 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.280480 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/258e5cf7-89af-4f93-8423-24e4897e66e6-istiod-ca-cert\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.280623 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.280511 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-workload-certs\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.280623 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.280579 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-podinfo\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.280991 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.280628 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-workload-socket\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.280991 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.280782 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-data\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.280991 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.280874 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-credential-socket\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.280991 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.280973 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-workload-socket\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.281217 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.281030 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-workload-certs\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.281399 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.281372 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/258e5cf7-89af-4f93-8423-24e4897e66e6-istiod-ca-cert\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.282849 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.282831 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-envoy\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.283192 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.283172 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-podinfo\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.300885 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.300850 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-token\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.301011 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.300985 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj5dd\" (UniqueName: \"kubernetes.io/projected/258e5cf7-89af-4f93-8423-24e4897e66e6-kube-api-access-cj5dd\") pod \"e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.458449 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.458365 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:40.494027 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.493996 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:40.748576 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.748549 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d"] Oct 30 22:02:40.750873 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:02:40.750848 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258e5cf7_89af_4f93_8423_24e4897e66e6.slice/crio-d7ed780618dfb84679c9098205db189d9ceb6051047acbe358c9b28535feaaba WatchSource:0}: Error finding container d7ed780618dfb84679c9098205db189d9ceb6051047acbe358c9b28535feaaba: Status 404 returned error can't find the container with id d7ed780618dfb84679c9098205db189d9ceb6051047acbe358c9b28535feaaba Oct 30 22:02:40.752985 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.752954 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:02:40.753047 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.753018 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:02:40.753092 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:40.753049 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:02:41.270809 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:41.270773 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" event={"ID":"258e5cf7-89af-4f93-8423-24e4897e66e6","Type":"ContainerStarted","Data":"5d5a06a288aa7ae080b81d52697937159439d93dd1d485deb72c833e3bf0fa1b"} Oct 30 22:02:41.270991 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:41.270854 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" event={"ID":"258e5cf7-89af-4f93-8423-24e4897e66e6","Type":"ContainerStarted","Data":"d7ed780618dfb84679c9098205db189d9ceb6051047acbe358c9b28535feaaba"} Oct 30 22:02:41.298334 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:41.298288 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" podStartSLOduration=1.298275473 podStartE2EDuration="1.298275473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:41.295985384 +0000 UTC m=+2029.891802044" watchObservedRunningTime="2026-10-30 22:02:41.298275473 +0000 UTC m=+2029.894092133" Oct 30 22:02:41.459324 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:41.459286 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:41.464447 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:41.464423 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:42.275818 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:42.275786 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:42.276851 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:42.276794 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:02:43.595659 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:43.595622 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:43.698984 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:43.698942 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:44.813197 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:44.813073 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:44.878491 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:44.878456 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz"] Oct 30 22:02:44.882079 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:44.882057 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz" Oct 30 22:02:44.899056 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:44.898796 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz"] Oct 30 22:02:44.900589 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:44.899379 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-fa15d4\"" Oct 30 22:02:44.900721 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:44.900677 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-17a1ad9a-dockercfg-g69b8\"" Oct 30 22:02:44.913161 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:44.913123 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh"] Oct 30 22:02:44.917294 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:44.917263 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh" Oct 30 22:02:44.928273 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:44.928241 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a1ae2e-9dfb-48c7-829f-ee129d480d56-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz\" (UID: \"21a1ae2e-9dfb-48c7-829f-ee129d480d56\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz" Oct 30 22:02:44.928442 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:44.928400 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwsdx\" (UniqueName: \"kubernetes.io/projected/21a1ae2e-9dfb-48c7-829f-ee129d480d56-kube-api-access-jwsdx\") pod \"payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz\" (UID: \"21a1ae2e-9dfb-48c7-829f-ee129d480d56\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz" Oct 30 22:02:44.936815 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:44.935884 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh"] Oct 30 22:02:44.952783 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:44.952648 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:45.014588 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.014560 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 22:02:45.018007 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.017956 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 22:02:45.022081 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.022060 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 22:02:45.029478 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.029452 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwsdx\" (UniqueName: \"kubernetes.io/projected/21a1ae2e-9dfb-48c7-829f-ee129d480d56-kube-api-access-jwsdx\") pod \"payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz\" (UID: \"21a1ae2e-9dfb-48c7-829f-ee129d480d56\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz" Oct 30 22:02:45.029630 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.029487 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b7f8a19-123d-4368-8b73-10361bc06a26-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh\" (UID: \"7b7f8a19-123d-4368-8b73-10361bc06a26\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh" Oct 30 22:02:45.029630 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.029508 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqbdl\" (UniqueName: \"kubernetes.io/projected/7b7f8a19-123d-4368-8b73-10361bc06a26-kube-api-access-vqbdl\") pod \"payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh\" (UID: \"7b7f8a19-123d-4368-8b73-10361bc06a26\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh" Oct 30 22:02:45.029806 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.029681 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a1ae2e-9dfb-48c7-829f-ee129d480d56-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz\" (UID: \"21a1ae2e-9dfb-48c7-829f-ee129d480d56\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz" Oct 30 22:02:45.030360 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.030342 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a1ae2e-9dfb-48c7-829f-ee129d480d56-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz\" (UID: \"21a1ae2e-9dfb-48c7-829f-ee129d480d56\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz" Oct 30 22:02:45.046922 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.046892 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwsdx\" (UniqueName: \"kubernetes.io/projected/21a1ae2e-9dfb-48c7-829f-ee129d480d56-kube-api-access-jwsdx\") pod \"payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz\" (UID: \"21a1ae2e-9dfb-48c7-829f-ee129d480d56\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz" Oct 30 22:02:45.130617 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.130577 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-data\") pod \"79d7a2c8-0585-48a3-88d4-d49b23672216\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " Oct 30 22:02:45.130617 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.130619 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-workload-socket\") pod \"79d7a2c8-0585-48a3-88d4-d49b23672216\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " Oct 30 22:02:45.131533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.130651 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-token\" (UniqueName: \"kubernetes.io/projected/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-token\") pod \"79d7a2c8-0585-48a3-88d4-d49b23672216\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " Oct 30 22:02:45.131533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.130681 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-credential-socket\") pod \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " Oct 30 22:02:45.131533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.130721 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-podinfo\") pod \"79d7a2c8-0585-48a3-88d4-d49b23672216\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " Oct 30 22:02:45.131533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.130780 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-data\") pod \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " Oct 30 22:02:45.131533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.130805 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istiod-ca-cert\") pod \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " Oct 30 22:02:45.131533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.130836 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-envoy\") pod \"79d7a2c8-0585-48a3-88d4-d49b23672216\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " Oct 30 22:02:45.131533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.130907 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-token\" (UniqueName: \"kubernetes.io/projected/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-token\") pod \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " Oct 30 22:02:45.131533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.130941 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-workload-socket\") pod \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " Oct 30 22:02:45.131533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.130967 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-podinfo\") pod \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " Oct 30 22:02:45.131533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.130971 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-credential-socket" pod "493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b" (UID: "493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:45.131533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.131002 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-data" pod "79d7a2c8-0585-48a3-88d4-d49b23672216" (UID: "79d7a2c8-0585-48a3-88d4-d49b23672216"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:45.131533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.131007 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-workload-certs\") pod \"79d7a2c8-0585-48a3-88d4-d49b23672216\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " Oct 30 22:02:45.131533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.131080 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-data" pod "9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9" (UID: "9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:45.131533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.131108 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d7a2c8-0585-48a3-88d4-d49b23672216-kube-api-access-glp8f\" (UniqueName: \"kubernetes.io/projected/79d7a2c8-0585-48a3-88d4-d49b23672216-kube-api-access-glp8f\") pod \"79d7a2c8-0585-48a3-88d4-d49b23672216\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " Oct 30 22:02:45.131533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.131149 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-credential-socket\") pod \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " Oct 30 22:02:45.131533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.131180 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-workload-socket\") pod \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " Oct 30 22:02:45.131533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.131201 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-workload-certs" pod "79d7a2c8-0585-48a3-88d4-d49b23672216" (UID: "79d7a2c8-0585-48a3-88d4-d49b23672216"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:45.131533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.131209 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-envoy\") pod \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " Oct 30 22:02:45.131533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.131224 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-workload-socket" pod "493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b" (UID: "493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:45.131533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.131386 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-workload-certs\") pod \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " Oct 30 22:02:45.131533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.131428 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-credential-socket" pod "9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9" (UID: "9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:45.131533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.131477 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-credential-socket\") pod \"79d7a2c8-0585-48a3-88d4-d49b23672216\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " Oct 30 22:02:45.131533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.131527 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-token\" (UniqueName: \"kubernetes.io/projected/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-token\") pod \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " Oct 30 22:02:45.132522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.131559 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-envoy\") pod \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " Oct 30 22:02:45.132522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.131592 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istiod-ca-cert\") pod \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " Oct 30 22:02:45.132522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.131624 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-kube-api-access-xgjl5\" (UniqueName: \"kubernetes.io/projected/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-kube-api-access-xgjl5\") pod \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " Oct 30 22:02:45.132522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.131638 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-workload-socket" pod "9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9" (UID: "9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:45.132522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.131676 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-workload-certs\") pod \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " Oct 30 22:02:45.132522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.131715 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-data\") pod \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " Oct 30 22:02:45.132522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.131774 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-kube-api-access-7zxs5\" (UniqueName: \"kubernetes.io/projected/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-kube-api-access-7zxs5\") pod \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\" (UID: \"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b\") " Oct 30 22:02:45.132522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.131809 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d7a2c8-0585-48a3-88d4-d49b23672216-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d7a2c8-0585-48a3-88d4-d49b23672216-istiod-ca-cert\") pod \"79d7a2c8-0585-48a3-88d4-d49b23672216\" (UID: \"79d7a2c8-0585-48a3-88d4-d49b23672216\") " Oct 30 22:02:45.132522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.131843 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-podinfo\") pod \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\" (UID: \"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9\") " Oct 30 22:02:45.132522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.131933 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istiod-ca-cert" pod "9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9" (UID: "9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:45.132522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.131953 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-workload-socket" pod "79d7a2c8-0585-48a3-88d4-d49b23672216" (UID: "79d7a2c8-0585-48a3-88d4-d49b23672216"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:45.132522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.132062 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b7f8a19-123d-4368-8b73-10361bc06a26-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh\" (UID: \"7b7f8a19-123d-4368-8b73-10361bc06a26\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh" Oct 30 22:02:45.132522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.132092 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqbdl\" (UniqueName: \"kubernetes.io/projected/7b7f8a19-123d-4368-8b73-10361bc06a26-kube-api-access-vqbdl\") pod \"payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh\" (UID: \"7b7f8a19-123d-4368-8b73-10361bc06a26\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh" Oct 30 22:02:45.132522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.132260 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istiod-ca-cert" pod "493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b" (UID: "493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:45.132522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.132360 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.132522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.132376 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.132522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.132389 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.132522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.132402 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.132522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.132416 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.132522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.132430 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.132522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.132443 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.132522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.132457 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.132522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.132470 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.132522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.132484 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.134222 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.134132 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-workload-certs" pod "9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9" (UID: "9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:45.134222 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.134178 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d7a2c8-0585-48a3-88d4-d49b23672216-istiod-ca-cert" pod "79d7a2c8-0585-48a3-88d4-d49b23672216" (UID: "79d7a2c8-0585-48a3-88d4-d49b23672216"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:45.134494 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.134455 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-credential-socket" pod "79d7a2c8-0585-48a3-88d4-d49b23672216" (UID: "79d7a2c8-0585-48a3-88d4-d49b23672216"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:45.134743 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.134607 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-data" pod "493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b" (UID: "493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:45.134743 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.134621 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-workload-certs" pod "493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b" (UID: "493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:45.135171 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.135097 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b7f8a19-123d-4368-8b73-10361bc06a26-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh\" (UID: \"7b7f8a19-123d-4368-8b73-10361bc06a26\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh" Oct 30 22:02:45.136241 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.136204 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d7a2c8-0585-48a3-88d4-d49b23672216-kube-api-access-glp8f" pod "79d7a2c8-0585-48a3-88d4-d49b23672216" (UID: "79d7a2c8-0585-48a3-88d4-d49b23672216"). InnerVolumeSpecName "kube-api-access-glp8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:45.136241 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.136210 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-envoy" pod "493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b" (UID: "493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:45.136501 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.136361 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-podinfo" pod "493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b" (UID: "493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:02:45.136501 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.136448 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-token" pod "79d7a2c8-0585-48a3-88d4-d49b23672216" (UID: "79d7a2c8-0585-48a3-88d4-d49b23672216"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:45.136958 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.136686 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-token" pod "493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b" (UID: "493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:45.136958 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.136824 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-token" pod "9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9" (UID: "9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:45.136958 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.136926 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-podinfo" pod "9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9" (UID: "9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:02:45.137122 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.137077 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-kube-api-access-xgjl5" pod "9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9" (UID: "9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9"). InnerVolumeSpecName "kube-api-access-xgjl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:45.137164 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.137112 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-envoy" pod "79d7a2c8-0585-48a3-88d4-d49b23672216" (UID: "79d7a2c8-0585-48a3-88d4-d49b23672216"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:45.137248 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.137232 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-kube-api-access-7zxs5" pod "493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b" (UID: "493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b"). InnerVolumeSpecName "kube-api-access-7zxs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:45.137576 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.137554 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-envoy" pod "9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9" (UID: "9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:45.138025 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.137998 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-podinfo" pod "79d7a2c8-0585-48a3-88d4-d49b23672216" (UID: "79d7a2c8-0585-48a3-88d4-d49b23672216"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:02:45.147832 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.147806 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqbdl\" (UniqueName: \"kubernetes.io/projected/7b7f8a19-123d-4368-8b73-10361bc06a26-kube-api-access-vqbdl\") pod \"payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh\" (UID: \"7b7f8a19-123d-4368-8b73-10361bc06a26\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh" Oct 30 22:02:45.195930 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.195889 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz" Oct 30 22:02:45.233999 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.233963 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glp8f\" (UniqueName: \"kubernetes.io/projected/79d7a2c8-0585-48a3-88d4-d49b23672216-kube-api-access-glp8f\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.233999 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.233996 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.233999 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.234007 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.234242 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.234016 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.234242 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.234025 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.234242 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.234036 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.234242 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.234044 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgjl5\" (UniqueName: \"kubernetes.io/projected/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-kube-api-access-xgjl5\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.234242 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.234054 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.234242 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.234063 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.234242 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.234084 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zxs5\" (UniqueName: \"kubernetes.io/projected/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-kube-api-access-7zxs5\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.234242 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.234094 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d7a2c8-0585-48a3-88d4-d49b23672216-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.234242 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.234104 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.234242 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.234113 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.234242 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.234122 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.234242 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.234131 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d7a2c8-0585-48a3-88d4-d49b23672216-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.234242 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.234139 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.234242 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.234148 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.240796 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.240744 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh" Oct 30 22:02:45.294167 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.294130 5801 generic.go:356] "Generic (PLEG): container finished" podID="493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b" containerID="7556d3f99a0c67aa563c1911fa1b53ebbb0775b1569160dd71a41f9c8b83bcca" exitCode=0 Oct 30 22:02:45.294325 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.294186 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" Oct 30 22:02:45.294325 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.294208 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" event={"ID":"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b","Type":"ContainerDied","Data":"7556d3f99a0c67aa563c1911fa1b53ebbb0775b1569160dd71a41f9c8b83bcca"} Oct 30 22:02:45.294325 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.294262 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr" event={"ID":"493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b","Type":"ContainerDied","Data":"76e5a25cc766c804cc6bbcaa5caca50ee45e5faf2855f62e804e476fc69b42a5"} Oct 30 22:02:45.294325 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.294295 5801 scope.go:122] "RemoveContainer" containerID="7556d3f99a0c67aa563c1911fa1b53ebbb0775b1569160dd71a41f9c8b83bcca" Oct 30 22:02:45.297261 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.297222 5801 generic.go:356] "Generic (PLEG): container finished" podID="9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9" containerID="491aa761acede556be776655a81f800a3967ef7b917a11436202c5d2243e9e99" exitCode=0 Oct 30 22:02:45.297261 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.297256 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" event={"ID":"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9","Type":"ContainerDied","Data":"491aa761acede556be776655a81f800a3967ef7b917a11436202c5d2243e9e99"} Oct 30 22:02:45.297474 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.297318 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" Oct 30 22:02:45.297474 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.297322 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm" event={"ID":"9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9","Type":"ContainerDied","Data":"b7cdc347562175f31758816c333e3ea3b0c23ac5a4acd661b54ca147525c64d1"} Oct 30 22:02:45.299393 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.299370 5801 generic.go:356] "Generic (PLEG): container finished" podID="79d7a2c8-0585-48a3-88d4-d49b23672216" containerID="8eeff9e295780e2ce15ac0f0d42f372d5844b94d28b2a1d5bdc026100dad48d5" exitCode=0 Oct 30 22:02:45.299516 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.299415 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" Oct 30 22:02:45.299516 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.299434 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" event={"ID":"79d7a2c8-0585-48a3-88d4-d49b23672216","Type":"ContainerDied","Data":"8eeff9e295780e2ce15ac0f0d42f372d5844b94d28b2a1d5bdc026100dad48d5"} Oct 30 22:02:45.299516 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.299486 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk" event={"ID":"79d7a2c8-0585-48a3-88d4-d49b23672216","Type":"ContainerDied","Data":"211580f9c8bdab1d35270f9f520a86742462f21574f6f2039066261a78b93f62"} Oct 30 22:02:45.319311 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.318990 5801 scope.go:122] "RemoveContainer" containerID="7556d3f99a0c67aa563c1911fa1b53ebbb0775b1569160dd71a41f9c8b83bcca" Oct 30 22:02:45.319426 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:02:45.319384 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7556d3f99a0c67aa563c1911fa1b53ebbb0775b1569160dd71a41f9c8b83bcca\": container with ID starting with 7556d3f99a0c67aa563c1911fa1b53ebbb0775b1569160dd71a41f9c8b83bcca not found: ID does not exist" containerID="7556d3f99a0c67aa563c1911fa1b53ebbb0775b1569160dd71a41f9c8b83bcca" Oct 30 22:02:45.319474 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.319419 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7556d3f99a0c67aa563c1911fa1b53ebbb0775b1569160dd71a41f9c8b83bcca"} err="failed to get container status \"7556d3f99a0c67aa563c1911fa1b53ebbb0775b1569160dd71a41f9c8b83bcca\": rpc error: code = NotFound desc = could not find container \"7556d3f99a0c67aa563c1911fa1b53ebbb0775b1569160dd71a41f9c8b83bcca\": container with ID starting with 7556d3f99a0c67aa563c1911fa1b53ebbb0775b1569160dd71a41f9c8b83bcca not found: ID does not exist" Oct 30 22:02:45.319474 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.319444 5801 scope.go:122] "RemoveContainer" containerID="491aa761acede556be776655a81f800a3967ef7b917a11436202c5d2243e9e99" Oct 30 22:02:45.333311 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.333279 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr"] Oct 30 22:02:45.333435 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.333421 5801 scope.go:122] "RemoveContainer" containerID="491aa761acede556be776655a81f800a3967ef7b917a11436202c5d2243e9e99" Oct 30 22:02:45.333905 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:02:45.333874 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"491aa761acede556be776655a81f800a3967ef7b917a11436202c5d2243e9e99\": container with ID starting with 491aa761acede556be776655a81f800a3967ef7b917a11436202c5d2243e9e99 not found: ID does not exist" containerID="491aa761acede556be776655a81f800a3967ef7b917a11436202c5d2243e9e99" Oct 30 22:02:45.333997 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.333913 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491aa761acede556be776655a81f800a3967ef7b917a11436202c5d2243e9e99"} err="failed to get container status \"491aa761acede556be776655a81f800a3967ef7b917a11436202c5d2243e9e99\": rpc error: code = NotFound desc = could not find container \"491aa761acede556be776655a81f800a3967ef7b917a11436202c5d2243e9e99\": container with ID starting with 491aa761acede556be776655a81f800a3967ef7b917a11436202c5d2243e9e99 not found: ID does not exist" Oct 30 22:02:45.333997 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.333935 5801 scope.go:122] "RemoveContainer" containerID="8eeff9e295780e2ce15ac0f0d42f372d5844b94d28b2a1d5bdc026100dad48d5" Oct 30 22:02:45.341366 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.341329 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-z2lsr"] Oct 30 22:02:45.344822 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.344724 5801 scope.go:122] "RemoveContainer" containerID="63338caca66d5d40f5435f9815a1df944f421544469b85e1be47e43bd2b3733e" Oct 30 22:02:45.354524 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.354485 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm"] Oct 30 22:02:45.356024 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.356002 5801 scope.go:122] "RemoveContainer" containerID="8eeff9e295780e2ce15ac0f0d42f372d5844b94d28b2a1d5bdc026100dad48d5" Oct 30 22:02:45.356358 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:02:45.356337 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eeff9e295780e2ce15ac0f0d42f372d5844b94d28b2a1d5bdc026100dad48d5\": container with ID starting with 8eeff9e295780e2ce15ac0f0d42f372d5844b94d28b2a1d5bdc026100dad48d5 not found: ID does not exist" containerID="8eeff9e295780e2ce15ac0f0d42f372d5844b94d28b2a1d5bdc026100dad48d5" Oct 30 22:02:45.356461 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.356365 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eeff9e295780e2ce15ac0f0d42f372d5844b94d28b2a1d5bdc026100dad48d5"} err="failed to get container status \"8eeff9e295780e2ce15ac0f0d42f372d5844b94d28b2a1d5bdc026100dad48d5\": rpc error: code = NotFound desc = could not find container \"8eeff9e295780e2ce15ac0f0d42f372d5844b94d28b2a1d5bdc026100dad48d5\": container with ID starting with 8eeff9e295780e2ce15ac0f0d42f372d5844b94d28b2a1d5bdc026100dad48d5 not found: ID does not exist" Oct 30 22:02:45.356461 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.356386 5801 scope.go:122] "RemoveContainer" containerID="63338caca66d5d40f5435f9815a1df944f421544469b85e1be47e43bd2b3733e" Oct 30 22:02:45.356630 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:02:45.356600 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63338caca66d5d40f5435f9815a1df944f421544469b85e1be47e43bd2b3733e\": container with ID starting with 63338caca66d5d40f5435f9815a1df944f421544469b85e1be47e43bd2b3733e not found: ID does not exist" containerID="63338caca66d5d40f5435f9815a1df944f421544469b85e1be47e43bd2b3733e" Oct 30 22:02:45.356782 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.356636 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63338caca66d5d40f5435f9815a1df944f421544469b85e1be47e43bd2b3733e"} err="failed to get container status \"63338caca66d5d40f5435f9815a1df944f421544469b85e1be47e43bd2b3733e\": rpc error: code = NotFound desc = could not find container \"63338caca66d5d40f5435f9815a1df944f421544469b85e1be47e43bd2b3733e\": container with ID starting with 63338caca66d5d40f5435f9815a1df944f421544469b85e1be47e43bd2b3733e not found: ID does not exist" Oct 30 22:02:45.361451 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.361427 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jmcsm"] Oct 30 22:02:45.376206 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.376173 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk"] Oct 30 22:02:45.383659 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.383632 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-448hk"] Oct 30 22:02:45.480783 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.480564 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz"] Oct 30 22:02:45.522968 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.522939 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b" path="/var/lib/kubelet/pods/493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b/volumes" Oct 30 22:02:45.523443 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.523429 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79d7a2c8-0585-48a3-88d4-d49b23672216" path="/var/lib/kubelet/pods/79d7a2c8-0585-48a3-88d4-d49b23672216/volumes" Oct 30 22:02:45.523933 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.523910 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9" path="/var/lib/kubelet/pods/9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9/volumes" Oct 30 22:02:45.541026 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.540998 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh"] Oct 30 22:02:45.542925 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:02:45.542894 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b7f8a19_123d_4368_8b73_10361bc06a26.slice/crio-e039ec92aeb95809e3a099f4f02260b66ca4168e11ee9948736f7596c68b981b WatchSource:0}: Error finding container e039ec92aeb95809e3a099f4f02260b66ca4168e11ee9948736f7596c68b981b: Status 404 returned error can't find the container with id e039ec92aeb95809e3a099f4f02260b66ca4168e11ee9948736f7596c68b981b Oct 30 22:02:45.787827 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:45.787712 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:46.307701 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:46.307666 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz" event={"ID":"21a1ae2e-9dfb-48c7-829f-ee129d480d56","Type":"ContainerStarted","Data":"3fc90682231086322655f7f37b439464b0e6812da033ff2ba624aba8c3bc037d"} Oct 30 22:02:46.308187 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:46.307718 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz" event={"ID":"21a1ae2e-9dfb-48c7-829f-ee129d480d56","Type":"ContainerStarted","Data":"1f748a5f668b868ee73d2976a75946593895e67974d9ce6b773268afac3c2743"} Oct 30 22:02:46.308187 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:46.307874 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz" Oct 30 22:02:46.309261 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:46.309237 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh" event={"ID":"7b7f8a19-123d-4368-8b73-10361bc06a26","Type":"ContainerStarted","Data":"8d2127dd3644a8797fd48de09470101869a8cea901ff8c406d7f267be87e2710"} Oct 30 22:02:46.309391 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:46.309275 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh" event={"ID":"7b7f8a19-123d-4368-8b73-10361bc06a26","Type":"ContainerStarted","Data":"e039ec92aeb95809e3a099f4f02260b66ca4168e11ee9948736f7596c68b981b"} Oct 30 22:02:46.309391 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:46.309331 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh" Oct 30 22:02:46.327600 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:46.327547 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz" podStartSLOduration=2.327535124 podStartE2EDuration="2.327535124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:46.325903047 +0000 UTC m=+2034.921719725" watchObservedRunningTime="2026-10-30 22:02:46.327535124 +0000 UTC m=+2034.923351783" Oct 30 22:02:46.345015 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:46.344961 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh" podStartSLOduration=2.34494739 podStartE2EDuration="2.34494739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:46.343359968 +0000 UTC m=+2034.939176654" watchObservedRunningTime="2026-10-30 22:02:46.34494739 +0000 UTC m=+2034.940764047" Oct 30 22:02:46.586580 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:46.586483 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:47.310933 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:47.310898 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:48.168774 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:48.166518 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:48.873778 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:48.868543 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:49.679345 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:49.679305 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:50.431045 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:50.431008 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:51.079262 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:51.079218 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:51.778492 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:51.778447 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:52.482523 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:52.482478 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:53.223087 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:53.223035 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:53.986205 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:53.986113 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:54.711980 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.711942 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:54.712480 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.712392 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd"] Oct 30 22:02:54.712921 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.712895 5801 cpu_manager.go:414] "containerMap: removing container" podUID="9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9" containerName="istio-proxy" Oct 30 22:02:54.713063 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.712936 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9" containerName="istio-proxy" Oct 30 22:02:54.713063 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.712965 5801 cpu_manager.go:414] "containerMap: removing container" podUID="79d7a2c8-0585-48a3-88d4-d49b23672216" containerName="istio-proxy" Oct 30 22:02:54.713063 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.712973 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d7a2c8-0585-48a3-88d4-d49b23672216" containerName="istio-proxy" Oct 30 22:02:54.713063 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.712994 5801 cpu_manager.go:414] "containerMap: removing container" podUID="79d7a2c8-0585-48a3-88d4-d49b23672216" containerName="istio-proxy" Oct 30 22:02:54.713063 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.713002 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d7a2c8-0585-48a3-88d4-d49b23672216" containerName="istio-proxy" Oct 30 22:02:54.713063 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.713029 5801 cpu_manager.go:414] "containerMap: removing container" podUID="493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b" containerName="istio-proxy" Oct 30 22:02:54.713063 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.713036 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b" containerName="istio-proxy" Oct 30 22:02:54.713446 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.713137 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="493d5dd9-6bc8-43f2-b9ea-bf9989c20c2b" containerName="istio-proxy" Oct 30 22:02:54.713446 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.713151 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f8dce17-6c39-43d9-9a59-0f2cac4cb7e9" containerName="istio-proxy" Oct 30 22:02:54.713446 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.713189 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d7a2c8-0585-48a3-88d4-d49b23672216" containerName="istio-proxy" Oct 30 22:02:54.713446 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.713203 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d7a2c8-0585-48a3-88d4-d49b23672216" containerName="istio-proxy" Oct 30 22:02:54.717765 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.717734 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.721379 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.721353 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4"] Oct 30 22:02:54.725168 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.725146 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl"] Oct 30 22:02:54.725325 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.725261 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.728726 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.728706 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.732541 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.732510 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd"] Oct 30 22:02:54.740274 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.740237 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4"] Oct 30 22:02:54.743962 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.743930 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl"] Oct 30 22:02:54.841439 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.841395 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57fe57d2-19a5-4ef7-b606-7753b0b7d902-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.841439 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.841438 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2cfd5-7f5a-4624-a753-30bc982288d4-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.841735 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.841459 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57fe57d2-19a5-4ef7-b606-7753b0b7d902-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.841735 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.841536 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57fe57d2-19a5-4ef7-b606-7753b0b7d902-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.841735 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.841577 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/913cd5d1-0d49-41ba-921d-603a44b2126a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.841735 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.841672 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e2cfd5-7f5a-4624-a753-30bc982288d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.841735 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.841719 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57fe57d2-19a5-4ef7-b606-7753b0b7d902-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.842054 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.841805 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/913cd5d1-0d49-41ba-921d-603a44b2126a-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.842054 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.841828 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e2cfd5-7f5a-4624-a753-30bc982288d4-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.842054 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.841855 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gbl9\" (UniqueName: \"kubernetes.io/projected/913cd5d1-0d49-41ba-921d-603a44b2126a-kube-api-access-6gbl9\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.842054 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.841951 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/913cd5d1-0d49-41ba-921d-603a44b2126a-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.842054 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.841970 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/913cd5d1-0d49-41ba-921d-603a44b2126a-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.842054 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.842011 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57fe57d2-19a5-4ef7-b606-7753b0b7d902-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.842054 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.842050 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59zkc\" (UniqueName: \"kubernetes.io/projected/57fe57d2-19a5-4ef7-b606-7753b0b7d902-kube-api-access-59zkc\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.842440 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.842073 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/913cd5d1-0d49-41ba-921d-603a44b2126a-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.842440 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.842092 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e2cfd5-7f5a-4624-a753-30bc982288d4-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.842440 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.842107 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2cfd5-7f5a-4624-a753-30bc982288d4-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.842440 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.842125 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e2cfd5-7f5a-4624-a753-30bc982288d4-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.842440 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.842223 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8ljh\" (UniqueName: \"kubernetes.io/projected/c1e2cfd5-7f5a-4624-a753-30bc982288d4-kube-api-access-d8ljh\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.842440 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.842253 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/913cd5d1-0d49-41ba-921d-603a44b2126a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.842440 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.842273 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/913cd5d1-0d49-41ba-921d-603a44b2126a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.842440 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.842290 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e2cfd5-7f5a-4624-a753-30bc982288d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.842440 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.842332 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e2cfd5-7f5a-4624-a753-30bc982288d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.842440 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.842375 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57fe57d2-19a5-4ef7-b606-7753b0b7d902-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.843005 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.842447 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57fe57d2-19a5-4ef7-b606-7753b0b7d902-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.843005 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.842515 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57fe57d2-19a5-4ef7-b606-7753b0b7d902-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.843005 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.842537 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/913cd5d1-0d49-41ba-921d-603a44b2126a-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.943966 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.943921 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/913cd5d1-0d49-41ba-921d-603a44b2126a-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.943966 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.943958 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e2cfd5-7f5a-4624-a753-30bc982288d4-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.944232 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.943980 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2cfd5-7f5a-4624-a753-30bc982288d4-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.944232 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944090 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e2cfd5-7f5a-4624-a753-30bc982288d4-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.944232 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944137 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8ljh\" (UniqueName: \"kubernetes.io/projected/c1e2cfd5-7f5a-4624-a753-30bc982288d4-kube-api-access-d8ljh\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.944232 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944163 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/913cd5d1-0d49-41ba-921d-603a44b2126a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.944232 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944192 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/913cd5d1-0d49-41ba-921d-603a44b2126a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.944232 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944221 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e2cfd5-7f5a-4624-a753-30bc982288d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.944595 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944256 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e2cfd5-7f5a-4624-a753-30bc982288d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.944595 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944286 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57fe57d2-19a5-4ef7-b606-7753b0b7d902-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.944595 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944316 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57fe57d2-19a5-4ef7-b606-7753b0b7d902-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.944595 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944336 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e2cfd5-7f5a-4624-a753-30bc982288d4-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.944595 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944371 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57fe57d2-19a5-4ef7-b606-7753b0b7d902-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.944595 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944404 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/913cd5d1-0d49-41ba-921d-603a44b2126a-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.944595 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944412 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2cfd5-7f5a-4624-a753-30bc982288d4-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.944595 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944441 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57fe57d2-19a5-4ef7-b606-7753b0b7d902-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.944595 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944469 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2cfd5-7f5a-4624-a753-30bc982288d4-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.944595 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944500 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57fe57d2-19a5-4ef7-b606-7753b0b7d902-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.944595 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944527 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57fe57d2-19a5-4ef7-b606-7753b0b7d902-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.944595 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944551 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/913cd5d1-0d49-41ba-921d-603a44b2126a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.944595 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944589 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e2cfd5-7f5a-4624-a753-30bc982288d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.945362 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944630 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57fe57d2-19a5-4ef7-b606-7753b0b7d902-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.945362 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944665 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/913cd5d1-0d49-41ba-921d-603a44b2126a-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.945362 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944688 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e2cfd5-7f5a-4624-a753-30bc982288d4-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.945362 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944716 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gbl9\" (UniqueName: \"kubernetes.io/projected/913cd5d1-0d49-41ba-921d-603a44b2126a-kube-api-access-6gbl9\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.945362 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944787 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/913cd5d1-0d49-41ba-921d-603a44b2126a-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.945362 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944788 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57fe57d2-19a5-4ef7-b606-7753b0b7d902-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.945362 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944814 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/913cd5d1-0d49-41ba-921d-603a44b2126a-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.945362 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944845 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57fe57d2-19a5-4ef7-b606-7753b0b7d902-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.945362 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.944874 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59zkc\" (UniqueName: \"kubernetes.io/projected/57fe57d2-19a5-4ef7-b606-7753b0b7d902-kube-api-access-59zkc\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.945362 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.945041 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57fe57d2-19a5-4ef7-b606-7753b0b7d902-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.945362 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.945148 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e2cfd5-7f5a-4624-a753-30bc982288d4-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.946153 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.945413 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57fe57d2-19a5-4ef7-b606-7753b0b7d902-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.946153 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.945529 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2cfd5-7f5a-4624-a753-30bc982288d4-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.946153 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.945814 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57fe57d2-19a5-4ef7-b606-7753b0b7d902-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.946153 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.946059 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/913cd5d1-0d49-41ba-921d-603a44b2126a-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.946153 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.946063 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/913cd5d1-0d49-41ba-921d-603a44b2126a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.946487 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.946274 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/913cd5d1-0d49-41ba-921d-603a44b2126a-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.946487 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.946397 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e2cfd5-7f5a-4624-a753-30bc982288d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.946611 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.946493 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/913cd5d1-0d49-41ba-921d-603a44b2126a-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.946698 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.946667 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57fe57d2-19a5-4ef7-b606-7753b0b7d902-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.946826 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.946799 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/913cd5d1-0d49-41ba-921d-603a44b2126a-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.947213 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.947190 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e2cfd5-7f5a-4624-a753-30bc982288d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.947558 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.947528 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/913cd5d1-0d49-41ba-921d-603a44b2126a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.947558 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.947548 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/913cd5d1-0d49-41ba-921d-603a44b2126a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.948419 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.948401 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e2cfd5-7f5a-4624-a753-30bc982288d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.948693 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.948674 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57fe57d2-19a5-4ef7-b606-7753b0b7d902-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.948965 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.948952 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57fe57d2-19a5-4ef7-b606-7753b0b7d902-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.956839 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.956810 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/913cd5d1-0d49-41ba-921d-603a44b2126a-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:54.957415 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.957390 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57fe57d2-19a5-4ef7-b606-7753b0b7d902-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.957534 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.957389 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e2cfd5-7f5a-4624-a753-30bc982288d4-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.957534 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.957393 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8ljh\" (UniqueName: \"kubernetes.io/projected/c1e2cfd5-7f5a-4624-a753-30bc982288d4-kube-api-access-d8ljh\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-hqmk4\" (UID: \"c1e2cfd5-7f5a-4624-a753-30bc982288d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:54.957534 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.957501 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59zkc\" (UniqueName: \"kubernetes.io/projected/57fe57d2-19a5-4ef7-b606-7753b0b7d902-kube-api-access-59zkc\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-8fqrl\" (UID: \"57fe57d2-19a5-4ef7-b606-7753b0b7d902\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:54.957954 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:54.957933 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gbl9\" (UniqueName: \"kubernetes.io/projected/913cd5d1-0d49-41ba-921d-603a44b2126a-kube-api-access-6gbl9\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fm6jd\" (UID: \"913cd5d1-0d49-41ba-921d-603a44b2126a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:55.045450 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:55.045363 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:55.064966 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:55.064934 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:55.072318 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:55.072267 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:55.346520 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:55.346467 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd"] Oct 30 22:02:55.354604 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:02:55.354572 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod913cd5d1_0d49_41ba_921d_603a44b2126a.slice/crio-a8263354832402dbb874cb7f91a2d05c49cd4280f82cf041c9ae6c88a0bc247a WatchSource:0}: Error finding container a8263354832402dbb874cb7f91a2d05c49cd4280f82cf041c9ae6c88a0bc247a: Status 404 returned error can't find the container with id a8263354832402dbb874cb7f91a2d05c49cd4280f82cf041c9ae6c88a0bc247a Oct 30 22:02:55.357326 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:55.357292 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:02:55.357426 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:55.357378 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:02:55.357426 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:55.357407 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:02:55.376855 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:55.376719 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4"] Oct 30 22:02:55.379060 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:55.379028 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl"] Oct 30 22:02:55.383110 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:02:55.383080 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57fe57d2_19a5_4ef7_b606_7753b0b7d902.slice/crio-6a51b0ac8df62f983c633028ae826081444dfff16b51940b7c06bfef67641edf WatchSource:0}: Error finding container 6a51b0ac8df62f983c633028ae826081444dfff16b51940b7c06bfef67641edf: Status 404 returned error can't find the container with id 6a51b0ac8df62f983c633028ae826081444dfff16b51940b7c06bfef67641edf Oct 30 22:02:55.383896 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:02:55.383871 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e2cfd5_7f5a_4624_a753_30bc982288d4.slice/crio-140916e7d0e9c0478c6804cd5f5c84a9f72466b72da1d6526f942bb94eda2eb6 WatchSource:0}: Error finding container 140916e7d0e9c0478c6804cd5f5c84a9f72466b72da1d6526f942bb94eda2eb6: Status 404 returned error can't find the container with id 140916e7d0e9c0478c6804cd5f5c84a9f72466b72da1d6526f942bb94eda2eb6 Oct 30 22:02:55.385972 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:55.385935 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:02:55.386066 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:55.386003 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:02:55.386066 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:55.386039 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:02:55.386521 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:55.386384 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:02:55.386521 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:55.386456 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:02:55.386521 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:55.386499 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:02:55.472135 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:55.471660 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:56.179847 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:56.179738 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:56.407794 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:56.396928 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" event={"ID":"913cd5d1-0d49-41ba-921d-603a44b2126a","Type":"ContainerStarted","Data":"855c01997167dc2e3e0c9d8de7652596b8d25aa359ffe4b5bc3c8c1a350ec61a"} Oct 30 22:02:56.407794 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:56.397001 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" event={"ID":"913cd5d1-0d49-41ba-921d-603a44b2126a","Type":"ContainerStarted","Data":"a8263354832402dbb874cb7f91a2d05c49cd4280f82cf041c9ae6c88a0bc247a"} Oct 30 22:02:56.411533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:56.411494 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" event={"ID":"c1e2cfd5-7f5a-4624-a753-30bc982288d4","Type":"ContainerStarted","Data":"ce89a9c4acb4223f6b16aeeaa08558d43c1b26d58213b48ad80ab4997435dbb4"} Oct 30 22:02:56.411813 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:56.411792 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" event={"ID":"c1e2cfd5-7f5a-4624-a753-30bc982288d4","Type":"ContainerStarted","Data":"140916e7d0e9c0478c6804cd5f5c84a9f72466b72da1d6526f942bb94eda2eb6"} Oct 30 22:02:56.420778 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:56.419200 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" event={"ID":"57fe57d2-19a5-4ef7-b606-7753b0b7d902","Type":"ContainerStarted","Data":"3061ddfaaefce86bf65dd71a3bf10d994c428aad82768dc5f5b583f7259fa275"} Oct 30 22:02:56.420778 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:56.419260 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" event={"ID":"57fe57d2-19a5-4ef7-b606-7753b0b7d902","Type":"ContainerStarted","Data":"6a51b0ac8df62f983c633028ae826081444dfff16b51940b7c06bfef67641edf"} Oct 30 22:02:56.439776 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:56.439321 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" podStartSLOduration=2.439304928 podStartE2EDuration="2.439304928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:56.437083055 +0000 UTC m=+2045.032899738" watchObservedRunningTime="2026-10-30 22:02:56.439304928 +0000 UTC m=+2045.035121589" Oct 30 22:02:56.474805 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:56.473704 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" podStartSLOduration=2.473682321 podStartE2EDuration="2.473682321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:56.471589884 +0000 UTC m=+2045.067406597" watchObservedRunningTime="2026-10-30 22:02:56.473682321 +0000 UTC m=+2045.069498983" Oct 30 22:02:56.529785 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:56.508533 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" podStartSLOduration=2.508514151 podStartE2EDuration="2.508514151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:56.50579211 +0000 UTC m=+2045.101608773" watchObservedRunningTime="2026-10-30 22:02:56.508514151 +0000 UTC m=+2045.104330812" Oct 30 22:02:56.895779 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:56.895395 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:57.045585 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:57.045545 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:02:57.046718 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:57.046690 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Oct 30 22:02:57.046808 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:57.046746 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" podUID="913cd5d1-0d49-41ba-921d-603a44b2126a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Oct 30 22:02:57.066204 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:57.066119 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:02:57.076810 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:57.076222 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Oct 30 22:02:57.076810 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:57.076283 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" podUID="c1e2cfd5-7f5a-4624-a753-30bc982288d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Oct 30 22:02:57.076810 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:57.076554 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:02:57.076810 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:57.076689 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 30 22:02:57.076810 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:57.076721 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" podUID="57fe57d2-19a5-4ef7-b606-7753b0b7d902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 30 22:02:57.329647 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:57.328929 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh" Oct 30 22:02:57.330208 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:57.329970 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz" Oct 30 22:02:57.732867 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:57.710775 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:58.046172 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:58.046087 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Oct 30 22:02:58.046172 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:58.046140 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" podUID="913cd5d1-0d49-41ba-921d-603a44b2126a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Oct 30 22:02:58.065388 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:58.065346 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Oct 30 22:02:58.065709 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:58.065661 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" podUID="c1e2cfd5-7f5a-4624-a753-30bc982288d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Oct 30 22:02:58.074294 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:58.074260 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 30 22:02:58.074542 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:58.074509 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" podUID="57fe57d2-19a5-4ef7-b606-7753b0b7d902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 30 22:02:58.334463 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:58.333482 5801 patch_prober.go:29] interesting pod/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:02:58.334463 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:58.334116 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" podUID="258e5cf7-89af-4f93-8423-24e4897e66e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:02:58.486777 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:58.486664 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:58.697787 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:58.695628 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:58.825782 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:58.824232 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:59.046628 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:59.046580 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Oct 30 22:02:59.046884 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:59.046644 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" podUID="913cd5d1-0d49-41ba-921d-603a44b2126a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Oct 30 22:02:59.068304 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:59.068169 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Oct 30 22:02:59.068304 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:59.068233 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" podUID="c1e2cfd5-7f5a-4624-a753-30bc982288d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Oct 30 22:02:59.074238 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:59.074126 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 30 22:02:59.074238 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:59.074183 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" podUID="57fe57d2-19a5-4ef7-b606-7753b0b7d902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 30 22:02:59.275512 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:59.275422 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:59.396284 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:59.396165 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:59.463650 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:59.457013 5801 generic.go:356] "Generic (PLEG): container finished" podID="258e5cf7-89af-4f93-8423-24e4897e66e6" containerID="5d5a06a288aa7ae080b81d52697937159439d93dd1d485deb72c833e3bf0fa1b" exitCode=0 Oct 30 22:02:59.463650 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:59.457205 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" event={"ID":"258e5cf7-89af-4f93-8423-24e4897e66e6","Type":"ContainerDied","Data":"5d5a06a288aa7ae080b81d52697937159439d93dd1d485deb72c833e3bf0fa1b"} Oct 30 22:02:59.463650 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:59.457663 5801 scope.go:122] "RemoveContainer" containerID="5d5a06a288aa7ae080b81d52697937159439d93dd1d485deb72c833e3bf0fa1b" Oct 30 22:02:59.463650 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:59.461740 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:02:59.463650 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:59.461994 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:02:59.463650 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:59.462153 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:02:59.649139 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:02:59.649096 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:00.046364 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:00.046326 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Oct 30 22:03:00.046583 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:00.046387 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" podUID="913cd5d1-0d49-41ba-921d-603a44b2126a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Oct 30 22:03:00.073780 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:00.066217 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Oct 30 22:03:00.073780 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:00.066281 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" podUID="c1e2cfd5-7f5a-4624-a753-30bc982288d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Oct 30 22:03:00.080775 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:00.076219 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 30 22:03:00.080775 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:00.076286 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" podUID="57fe57d2-19a5-4ef7-b606-7753b0b7d902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 30 22:03:00.212817 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:00.206798 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:00.397778 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:03:00.397477 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:03:00.532808 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:00.532771 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" event={"ID":"258e5cf7-89af-4f93-8423-24e4897e66e6","Type":"ContainerStarted","Data":"9b5b87047fb640841099fc7fa194c18575a48e7f69ba7839f4905555500b9b69"} Oct 30 22:03:00.997778 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:00.988656 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:01.053788 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.053301 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Oct 30 22:03:01.053788 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.053380 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" podUID="913cd5d1-0d49-41ba-921d-603a44b2126a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Oct 30 22:03:01.072809 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.068252 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Oct 30 22:03:01.072809 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.068330 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" podUID="c1e2cfd5-7f5a-4624-a753-30bc982288d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Oct 30 22:03:01.076777 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.073811 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 30 22:03:01.076777 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.073884 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" podUID="57fe57d2-19a5-4ef7-b606-7753b0b7d902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 30 22:03:01.416777 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.401295 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s"] Oct 30 22:03:01.547971 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.491697 5801 patch_prober.go:29] interesting pod/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Oct 30 22:03:01.547971 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.491804 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" podUID="258e5cf7-89af-4f93-8423-24e4897e66e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Oct 30 22:03:01.547971 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.530518 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:01.683069 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.682981 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:03:01.683790 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.683769 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s"] Oct 30 22:03:01.684861 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.683674 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:03:01.723539 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.718478 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-fa15d4\"/\"simulator-single-main-fa15d4-kserve-self-signed-certs\"" Oct 30 22:03:01.723539 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.718651 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-fa15d4\"/\"default-dockercfg-kz4xk\"" Oct 30 22:03:01.723539 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.719444 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-fa15d4\"/\"kube-root-ca.crt\"" Oct 30 22:03:01.723539 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.719571 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-fa15d4\"/\"openshift-service-ca.crt\"" Oct 30 22:03:01.792787 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.792322 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:01.883777 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.883473 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbjl2\" (UniqueName: \"kubernetes.io/projected/a755d2ec-37af-488d-89ea-a4c0d0836bcd-kube-api-access-bbjl2\") pod \"simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s\" (UID: \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\") " pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:03:01.883777 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.883592 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-model-cache\") pod \"simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s\" (UID: \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\") " pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:03:01.883777 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.883638 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-tmp-dir\") pod \"simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s\" (UID: \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\") " pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:03:01.884136 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.883815 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a755d2ec-37af-488d-89ea-a4c0d0836bcd-tls-certs\") pod \"simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s\" (UID: \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\") " pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:03:01.884136 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.883931 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-dshm\") pod \"simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s\" (UID: \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\") " pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:03:01.884136 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.884026 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-home\") pod \"simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s\" (UID: \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\") " pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:03:01.985383 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.985273 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-home\") pod \"simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s\" (UID: \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\") " pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:03:01.986147 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.986122 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbjl2\" (UniqueName: \"kubernetes.io/projected/a755d2ec-37af-488d-89ea-a4c0d0836bcd-kube-api-access-bbjl2\") pod \"simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s\" (UID: \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\") " pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:03:01.986717 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.986695 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-model-cache\") pod \"simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s\" (UID: \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\") " pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:03:01.986916 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.986896 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-tmp-dir\") pod \"simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s\" (UID: \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\") " pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:03:01.987103 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.987084 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a755d2ec-37af-488d-89ea-a4c0d0836bcd-tls-certs\") pod \"simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s\" (UID: \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\") " pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:03:01.987254 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.987239 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-dshm\") pod \"simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s\" (UID: \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\") " pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:03:01.987382 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.987352 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-tmp-dir\") pod \"simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s\" (UID: \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\") " pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:03:01.987561 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.986034 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-home\") pod \"simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s\" (UID: \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\") " pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:03:01.987677 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.987115 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-model-cache\") pod \"simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s\" (UID: \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\") " pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:03:01.993463 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.993384 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a755d2ec-37af-488d-89ea-a4c0d0836bcd-tls-certs\") pod \"simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s\" (UID: \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\") " pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:03:01.994058 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:01.994027 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-dshm\") pod \"simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s\" (UID: \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\") " pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:03:02.000334 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:02.000299 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbjl2\" (UniqueName: \"kubernetes.io/projected/a755d2ec-37af-488d-89ea-a4c0d0836bcd-kube-api-access-bbjl2\") pod \"simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s\" (UID: \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\") " pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:03:02.046242 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:02.046203 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Oct 30 22:03:02.046462 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:02.046260 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" podUID="913cd5d1-0d49-41ba-921d-603a44b2126a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Oct 30 22:03:02.048508 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:02.048471 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:03:02.070776 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:02.065875 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Oct 30 22:03:02.070776 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:02.065955 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" podUID="c1e2cfd5-7f5a-4624-a753-30bc982288d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Oct 30 22:03:02.088798 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:02.079271 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 30 22:03:02.088798 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:02.079347 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" podUID="57fe57d2-19a5-4ef7-b606-7753b0b7d902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 30 22:03:02.401236 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:02.401100 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s"] Oct 30 22:03:02.459309 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:02.458871 5801 patch_prober.go:29] interesting pod/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Oct 30 22:03:02.459309 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:02.458929 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" podUID="258e5cf7-89af-4f93-8423-24e4897e66e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Oct 30 22:03:02.582814 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:02.580773 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:02.737933 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:02.737834 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" event={"ID":"a755d2ec-37af-488d-89ea-a4c0d0836bcd","Type":"ContainerStarted","Data":"9477589d9e16a36fd3b39f72f73f86868334e52a8d1e43faa3dff424e9dd0b7f"} Oct 30 22:03:02.737933 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:02.737892 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" event={"ID":"a755d2ec-37af-488d-89ea-a4c0d0836bcd","Type":"ContainerStarted","Data":"271cd5367487eec379ecff5dc960d08fb3e7a3bf36f447ad7ad6ba1a72920b95"} Oct 30 22:03:02.785778 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:02.779404 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" podStartSLOduration=1.7793844829999999 podStartE2EDuration="1.779384483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:02.779231243 +0000 UTC m=+2051.375047922" watchObservedRunningTime="2026-10-30 22:03:02.779384483 +0000 UTC m=+2051.375201144" Oct 30 22:03:03.046558 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:03.046463 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Oct 30 22:03:03.046558 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:03.046541 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" podUID="913cd5d1-0d49-41ba-921d-603a44b2126a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Oct 30 22:03:03.048620 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:03.048592 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:03:03.071783 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:03.069205 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Oct 30 22:03:03.071783 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:03.069284 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" podUID="c1e2cfd5-7f5a-4624-a753-30bc982288d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Oct 30 22:03:03.079775 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:03.075153 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 30 22:03:03.079775 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:03.075212 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" podUID="57fe57d2-19a5-4ef7-b606-7753b0b7d902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 30 22:03:03.326280 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:03.326239 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:03.463781 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:03.461255 5801 patch_prober.go:29] interesting pod/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Oct 30 22:03:03.463781 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:03.461332 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" podUID="258e5cf7-89af-4f93-8423-24e4897e66e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Oct 30 22:03:04.251525 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:04.251486 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:04.462975 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:04.462936 5801 patch_prober.go:29] interesting pod/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Oct 30 22:03:04.463304 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:04.463270 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" podUID="258e5cf7-89af-4f93-8423-24e4897e66e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Oct 30 22:03:04.995917 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:04.995868 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:03:05.009783 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:05.006879 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:03:05.009783 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:05.008859 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" Oct 30 22:03:05.044782 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:05.044205 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:03:05.044782 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:05.044656 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:03:05.045927 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:05.045892 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" Oct 30 22:03:05.112863 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:05.112823 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:03:05.113155 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:05.113126 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" podUID="57fe57d2-19a5-4ef7-b606-7753b0b7d902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:05.114141 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:05.114107 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:05.120522 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:05.120484 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:03:05.121447 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:05.121417 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:03:05.123448 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:05.123421 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" Oct 30 22:03:05.478261 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:05.478217 5801 patch_prober.go:29] interesting pod/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Oct 30 22:03:05.478544 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:05.478513 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" podUID="258e5cf7-89af-4f93-8423-24e4897e66e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Oct 30 22:03:05.977884 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:05.977840 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:06.477230 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:06.476292 5801 patch_prober.go:29] interesting pod/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Oct 30 22:03:06.477230 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:06.476355 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" podUID="258e5cf7-89af-4f93-8423-24e4897e66e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Oct 30 22:03:06.788502 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:06.788412 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:07.462620 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:07.462590 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:03:07.462968 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:07.462950 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:03:07.463779 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:07.463732 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:03:07.488972 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:07.488934 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:08.293891 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:08.293854 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:09.099250 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:09.099217 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:10.040960 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:10.040923 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:10.677684 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:10.677650 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:11.386904 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:11.386869 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:12.049524 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:12.049480 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:03:12.062311 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:12.062284 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:03:12.170661 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:12.170626 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:12.828392 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:12.828359 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:03:12.991059 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:12.991026 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:13.781963 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:13.781929 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:14.481933 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:14.481897 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:15.183911 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:15.183867 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:15.906233 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:15.906199 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:16.682936 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:16.682895 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:17.478091 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:17.478039 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:18.173456 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:18.173413 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:18.980477 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:18.980438 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:19.873016 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:19.872979 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:20.688843 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:20.688805 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:21.377943 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:21.377900 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:22.071007 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:22.070967 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:22.878280 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:22.878244 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:23.584380 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:23.584343 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:24.274158 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:24.274120 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:24.975899 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:24.975858 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:25.678381 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:25.678347 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:26.379836 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:26.379802 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:27.078672 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:27.078639 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:27.877142 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:27.877104 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:28.584853 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:28.584806 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:29.298806 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:29.298773 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:30.097456 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:30.097413 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:30.277126 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:03:30.277084 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:03:30.996777 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:30.996714 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:31.695395 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:31.695294 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:32.397248 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:32.397217 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:33.110942 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:33.110905 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38e5507c-n8zhl"] Oct 30 22:03:33.115665 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:33.115645 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e5507c-n8zhl" Oct 30 22:03:33.123511 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:33.123480 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:33.123626 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:33.123604 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 22:03:33.123783 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:33.123688 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:03:33.123871 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:33.123794 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:03:33.125182 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:33.125158 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38e5507c-n8zhl"] Oct 30 22:03:33.187168 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:33.187130 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:33.245747 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:33.245712 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr2wm\" (UniqueName: \"kubernetes.io/projected/5ba6a329-a55c-4567-a09d-194f79b2b09b-kube-api-access-zr2wm\") pod \"maas-api-revoke-sub-38e5507c-n8zhl\" (UID: \"5ba6a329-a55c-4567-a09d-194f79b2b09b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e5507c-n8zhl" Oct 30 22:03:33.245980 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:33.245890 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba6a329-a55c-4567-a09d-194f79b2b09b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38e5507c-n8zhl\" (UID: \"5ba6a329-a55c-4567-a09d-194f79b2b09b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e5507c-n8zhl" Oct 30 22:03:33.245980 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:33.245945 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba6a329-a55c-4567-a09d-194f79b2b09b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38e5507c-n8zhl\" (UID: \"5ba6a329-a55c-4567-a09d-194f79b2b09b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e5507c-n8zhl" Oct 30 22:03:33.346845 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:33.346809 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr2wm\" (UniqueName: \"kubernetes.io/projected/5ba6a329-a55c-4567-a09d-194f79b2b09b-kube-api-access-zr2wm\") pod \"maas-api-revoke-sub-38e5507c-n8zhl\" (UID: \"5ba6a329-a55c-4567-a09d-194f79b2b09b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e5507c-n8zhl" Oct 30 22:03:33.347065 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:33.346911 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba6a329-a55c-4567-a09d-194f79b2b09b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38e5507c-n8zhl\" (UID: \"5ba6a329-a55c-4567-a09d-194f79b2b09b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e5507c-n8zhl" Oct 30 22:03:33.347065 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:33.346950 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba6a329-a55c-4567-a09d-194f79b2b09b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38e5507c-n8zhl\" (UID: \"5ba6a329-a55c-4567-a09d-194f79b2b09b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e5507c-n8zhl" Oct 30 22:03:33.347570 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:33.347536 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba6a329-a55c-4567-a09d-194f79b2b09b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38e5507c-n8zhl\" (UID: \"5ba6a329-a55c-4567-a09d-194f79b2b09b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e5507c-n8zhl" Oct 30 22:03:33.349346 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:33.349317 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba6a329-a55c-4567-a09d-194f79b2b09b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38e5507c-n8zhl\" (UID: \"5ba6a329-a55c-4567-a09d-194f79b2b09b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e5507c-n8zhl" Oct 30 22:03:33.359693 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:33.359670 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr2wm\" (UniqueName: \"kubernetes.io/projected/5ba6a329-a55c-4567-a09d-194f79b2b09b-kube-api-access-zr2wm\") pod \"maas-api-revoke-sub-38e5507c-n8zhl\" (UID: \"5ba6a329-a55c-4567-a09d-194f79b2b09b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e5507c-n8zhl" Oct 30 22:03:33.431584 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:33.431508 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e5507c-n8zhl" Oct 30 22:03:33.706622 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:33.706596 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38e5507c-n8zhl"] Oct 30 22:03:33.708568 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:03:33.708538 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba6a329_a55c_4567_a09d_194f79b2b09b.slice/crio-fc8df7953e16f618ccc7cf6e5876d35cffab0f21999b8f29746b152be8928954 WatchSource:0}: Error finding container fc8df7953e16f618ccc7cf6e5876d35cffab0f21999b8f29746b152be8928954: Status 404 returned error can't find the container with id fc8df7953e16f618ccc7cf6e5876d35cffab0f21999b8f29746b152be8928954 Oct 30 22:03:33.930648 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:33.930609 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e5507c-n8zhl" event={"ID":"5ba6a329-a55c-4567-a09d-194f79b2b09b","Type":"ContainerStarted","Data":"fc8df7953e16f618ccc7cf6e5876d35cffab0f21999b8f29746b152be8928954"} Oct 30 22:03:33.980967 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:33.980938 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:34.684078 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:34.684040 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:34.938113 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:34.938015 5801 generic.go:356] "Generic (PLEG): container finished" podID="5ba6a329-a55c-4567-a09d-194f79b2b09b" containerID="2bbdc1fbafe5dab3d1da23debbb43a86958ea89dbf641c970a2606ecb9b47df6" exitCode=0 Oct 30 22:03:34.938288 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:34.938108 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e5507c-n8zhl" event={"ID":"5ba6a329-a55c-4567-a09d-194f79b2b09b","Type":"ContainerDied","Data":"2bbdc1fbafe5dab3d1da23debbb43a86958ea89dbf641c970a2606ecb9b47df6"} Oct 30 22:03:35.457113 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:35.457078 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:36.074518 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:36.074487 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e5507c-n8zhl" Oct 30 22:03:36.171442 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:36.171400 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:36.171763 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:36.171730 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ba6a329-a55c-4567-a09d-194f79b2b09b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba6a329-a55c-4567-a09d-194f79b2b09b-maas-api-xks-ca\") pod \"5ba6a329-a55c-4567-a09d-194f79b2b09b\" (UID: \"5ba6a329-a55c-4567-a09d-194f79b2b09b\") " Oct 30 22:03:36.171832 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:36.171809 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba6a329-a55c-4567-a09d-194f79b2b09b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba6a329-a55c-4567-a09d-194f79b2b09b-maas-api-openshift-ca\") pod \"5ba6a329-a55c-4567-a09d-194f79b2b09b\" (UID: \"5ba6a329-a55c-4567-a09d-194f79b2b09b\") " Oct 30 22:03:36.171946 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:36.171929 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba6a329-a55c-4567-a09d-194f79b2b09b-kube-api-access-zr2wm\" (UniqueName: \"kubernetes.io/projected/5ba6a329-a55c-4567-a09d-194f79b2b09b-kube-api-access-zr2wm\") pod \"5ba6a329-a55c-4567-a09d-194f79b2b09b\" (UID: \"5ba6a329-a55c-4567-a09d-194f79b2b09b\") " Oct 30 22:03:36.172176 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:36.172151 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba6a329-a55c-4567-a09d-194f79b2b09b-maas-api-openshift-ca" pod "5ba6a329-a55c-4567-a09d-194f79b2b09b" (UID: "5ba6a329-a55c-4567-a09d-194f79b2b09b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:36.173972 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:36.173938 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba6a329-a55c-4567-a09d-194f79b2b09b-kube-api-access-zr2wm" pod "5ba6a329-a55c-4567-a09d-194f79b2b09b" (UID: "5ba6a329-a55c-4567-a09d-194f79b2b09b"). InnerVolumeSpecName "kube-api-access-zr2wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:36.174068 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:36.173973 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ba6a329-a55c-4567-a09d-194f79b2b09b-maas-api-xks-ca" pod "5ba6a329-a55c-4567-a09d-194f79b2b09b" (UID: "5ba6a329-a55c-4567-a09d-194f79b2b09b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:36.272812 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:36.272726 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba6a329-a55c-4567-a09d-194f79b2b09b-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:36.272812 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:36.272779 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr2wm\" (UniqueName: \"kubernetes.io/projected/5ba6a329-a55c-4567-a09d-194f79b2b09b-kube-api-access-zr2wm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:36.272812 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:36.272792 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba6a329-a55c-4567-a09d-194f79b2b09b-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:36.869450 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:36.869411 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:36.952194 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:36.952159 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e5507c-n8zhl" event={"ID":"5ba6a329-a55c-4567-a09d-194f79b2b09b","Type":"ContainerDied","Data":"fc8df7953e16f618ccc7cf6e5876d35cffab0f21999b8f29746b152be8928954"} Oct 30 22:03:36.952348 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:36.952215 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc8df7953e16f618ccc7cf6e5876d35cffab0f21999b8f29746b152be8928954" Oct 30 22:03:36.952348 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:36.952172 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e5507c-n8zhl" Oct 30 22:03:37.580094 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:37.580060 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:38.280465 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:38.280433 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:38.980898 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:38.980858 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:39.780907 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:39.780871 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:40.590229 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:40.590176 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:41.275126 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:41.275089 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:42.082846 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:42.082797 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:42.779240 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:42.779180 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:43.505735 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:43.505686 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:43.989684 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:43.989639 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:44.278652 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:44.278576 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr"] Oct 30 22:03:44.279055 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:44.279037 5801 cpu_manager.go:414] "containerMap: removing container" podUID="5ba6a329-a55c-4567-a09d-194f79b2b09b" containerName="revoke-keys" Oct 30 22:03:44.279115 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:44.279056 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba6a329-a55c-4567-a09d-194f79b2b09b" containerName="revoke-keys" Oct 30 22:03:44.279166 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:44.279119 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba6a329-a55c-4567-a09d-194f79b2b09b" containerName="revoke-keys" Oct 30 22:03:44.282683 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:44.282665 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr" Oct 30 22:03:44.289193 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:44.289170 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:44.289609 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:44.289272 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 22:03:44.292303 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:44.292278 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr"] Oct 30 22:03:44.345199 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:44.345168 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67bf626-8615-46e2-8e35-7bb7009fc11b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr\" (UID: \"b67bf626-8615-46e2-8e35-7bb7009fc11b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr" Oct 30 22:03:44.345333 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:44.345204 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67bf626-8615-46e2-8e35-7bb7009fc11b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr\" (UID: \"b67bf626-8615-46e2-8e35-7bb7009fc11b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr" Oct 30 22:03:44.345333 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:44.345228 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jr4j\" (UniqueName: \"kubernetes.io/projected/b67bf626-8615-46e2-8e35-7bb7009fc11b-kube-api-access-4jr4j\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr\" (UID: \"b67bf626-8615-46e2-8e35-7bb7009fc11b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr" Oct 30 22:03:44.357488 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:44.357460 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:03:44.357587 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:44.357503 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:03:44.386687 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:44.386653 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:44.446022 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:44.445988 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67bf626-8615-46e2-8e35-7bb7009fc11b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr\" (UID: \"b67bf626-8615-46e2-8e35-7bb7009fc11b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr" Oct 30 22:03:44.446192 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:44.446075 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67bf626-8615-46e2-8e35-7bb7009fc11b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr\" (UID: \"b67bf626-8615-46e2-8e35-7bb7009fc11b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr" Oct 30 22:03:44.446192 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:44.446128 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jr4j\" (UniqueName: \"kubernetes.io/projected/b67bf626-8615-46e2-8e35-7bb7009fc11b-kube-api-access-4jr4j\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr\" (UID: \"b67bf626-8615-46e2-8e35-7bb7009fc11b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr" Oct 30 22:03:44.446790 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:44.446769 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67bf626-8615-46e2-8e35-7bb7009fc11b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr\" (UID: \"b67bf626-8615-46e2-8e35-7bb7009fc11b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr" Oct 30 22:03:44.448495 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:44.448473 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67bf626-8615-46e2-8e35-7bb7009fc11b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr\" (UID: \"b67bf626-8615-46e2-8e35-7bb7009fc11b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr" Oct 30 22:03:44.458705 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:44.458683 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jr4j\" (UniqueName: \"kubernetes.io/projected/b67bf626-8615-46e2-8e35-7bb7009fc11b-kube-api-access-4jr4j\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr\" (UID: \"b67bf626-8615-46e2-8e35-7bb7009fc11b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr" Oct 30 22:03:44.597023 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:44.597002 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr" Oct 30 22:03:44.870333 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:44.870288 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr"] Oct 30 22:03:44.872787 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:03:44.872728 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb67bf626_8615_46e2_8e35_7bb7009fc11b.slice/crio-1149c68d1baa24a8b065590a0a87b8ffbfa9c88224fb521947e9a8a668e9ea03 WatchSource:0}: Error finding container 1149c68d1baa24a8b065590a0a87b8ffbfa9c88224fb521947e9a8a668e9ea03: Status 404 returned error can't find the container with id 1149c68d1baa24a8b065590a0a87b8ffbfa9c88224fb521947e9a8a668e9ea03 Oct 30 22:03:44.993009 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:44.992985 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr" event={"ID":"b67bf626-8615-46e2-8e35-7bb7009fc11b","Type":"ContainerStarted","Data":"1149c68d1baa24a8b065590a0a87b8ffbfa9c88224fb521947e9a8a668e9ea03"} Oct 30 22:03:45.134259 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:45.134191 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:45.427225 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:45.427142 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:45.868828 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:45.868800 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:45.999865 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:45.999748 5801 generic.go:356] "Generic (PLEG): container finished" podID="b67bf626-8615-46e2-8e35-7bb7009fc11b" containerID="0df3de87c3265a833554562c399086ac584a16ffa87d3df4f14234ce89991261" exitCode=0 Oct 30 22:03:46.000024 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:45.999843 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr" event={"ID":"b67bf626-8615-46e2-8e35-7bb7009fc11b","Type":"ContainerDied","Data":"0df3de87c3265a833554562c399086ac584a16ffa87d3df4f14234ce89991261"} Oct 30 22:03:46.880348 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:46.880302 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:47.142988 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:47.142929 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr" Oct 30 22:03:47.172160 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:47.172131 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b67bf626-8615-46e2-8e35-7bb7009fc11b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67bf626-8615-46e2-8e35-7bb7009fc11b-maas-api-xks-ca\") pod \"b67bf626-8615-46e2-8e35-7bb7009fc11b\" (UID: \"b67bf626-8615-46e2-8e35-7bb7009fc11b\") " Oct 30 22:03:47.172281 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:47.172196 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b67bf626-8615-46e2-8e35-7bb7009fc11b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67bf626-8615-46e2-8e35-7bb7009fc11b-maas-api-openshift-ca\") pod \"b67bf626-8615-46e2-8e35-7bb7009fc11b\" (UID: \"b67bf626-8615-46e2-8e35-7bb7009fc11b\") " Oct 30 22:03:47.172281 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:47.172258 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b67bf626-8615-46e2-8e35-7bb7009fc11b-kube-api-access-4jr4j\" (UniqueName: \"kubernetes.io/projected/b67bf626-8615-46e2-8e35-7bb7009fc11b-kube-api-access-4jr4j\") pod \"b67bf626-8615-46e2-8e35-7bb7009fc11b\" (UID: \"b67bf626-8615-46e2-8e35-7bb7009fc11b\") " Oct 30 22:03:47.173013 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:47.172985 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b67bf626-8615-46e2-8e35-7bb7009fc11b-maas-api-openshift-ca" pod "b67bf626-8615-46e2-8e35-7bb7009fc11b" (UID: "b67bf626-8615-46e2-8e35-7bb7009fc11b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:47.174421 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:47.174398 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b67bf626-8615-46e2-8e35-7bb7009fc11b-kube-api-access-4jr4j" pod "b67bf626-8615-46e2-8e35-7bb7009fc11b" (UID: "b67bf626-8615-46e2-8e35-7bb7009fc11b"). InnerVolumeSpecName "kube-api-access-4jr4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:47.174609 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:47.174588 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b67bf626-8615-46e2-8e35-7bb7009fc11b-maas-api-xks-ca" pod "b67bf626-8615-46e2-8e35-7bb7009fc11b" (UID: "b67bf626-8615-46e2-8e35-7bb7009fc11b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:47.252699 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:47.252670 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:47.273328 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:47.273293 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jr4j\" (UniqueName: \"kubernetes.io/projected/b67bf626-8615-46e2-8e35-7bb7009fc11b-kube-api-access-4jr4j\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:47.273328 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:47.273329 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67bf626-8615-46e2-8e35-7bb7009fc11b-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:47.273328 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:47.273339 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67bf626-8615-46e2-8e35-7bb7009fc11b-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:47.685429 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:47.685392 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:48.012059 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:48.012027 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr" Oct 30 22:03:48.012530 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:48.012062 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr" event={"ID":"b67bf626-8615-46e2-8e35-7bb7009fc11b","Type":"ContainerDied","Data":"1149c68d1baa24a8b065590a0a87b8ffbfa9c88224fb521947e9a8a668e9ea03"} Oct 30 22:03:48.012530 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:48.012107 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1149c68d1baa24a8b065590a0a87b8ffbfa9c88224fb521947e9a8a668e9ea03" Oct 30 22:03:48.404789 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:48.404734 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:48.948114 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:48.948077 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s"] Oct 30 22:03:49.016180 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:49.016153 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:49.114599 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:49.114569 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:49.786602 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:49.786552 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:50.477615 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:50.477582 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:51.170495 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:51.170457 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:51.871563 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:51.871524 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:52.584619 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:52.584572 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:53.282212 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:53.282173 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:54.089431 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:54.089399 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:54.396594 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:54.396516 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh"] Oct 30 22:03:54.408377 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:54.408351 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz"] Oct 30 22:03:54.475615 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:54.475576 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:54.575427 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:54.575396 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:54.678925 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:54.678843 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:54.833393 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:54.833357 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:54.898915 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:54.898879 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr"] Oct 30 22:03:54.901960 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:54.901933 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fa15d4-pnvhr"] Oct 30 22:03:54.933601 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:54.933492 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:55.439272 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:55.439238 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:55.522489 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:55.522462 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b67bf626-8615-46e2-8e35-7bb7009fc11b" path="/var/lib/kubelet/pods/b67bf626-8615-46e2-8e35-7bb7009fc11b/volumes" Oct 30 22:03:58.886955 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:58.886916 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:59.397128 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:59.397091 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh" podUID="7b7f8a19-123d-4368-8b73-10361bc06a26" containerName="payload-processing" containerID="cri-o://8d2127dd3644a8797fd48de09470101869a8cea901ff8c406d7f267be87e2710" gracePeriod=25 Oct 30 22:03:59.409477 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:03:59.409443 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz" podUID="21a1ae2e-9dfb-48c7-829f-ee129d480d56" containerName="payload-pre-processing" containerID="cri-o://3fc90682231086322655f7f37b439464b0e6812da033ff2ba624aba8c3bc037d" gracePeriod=25 Oct 30 22:04:00.280766 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:04:00.280710 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:04:01.738929 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:01.738884 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:01.838552 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:01.838518 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:02.111685 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:02.111651 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d"] Oct 30 22:04:02.112221 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:02.112191 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" podUID="258e5cf7-89af-4f93-8423-24e4897e66e6" containerName="istio-proxy" containerID="cri-o://9b5b87047fb640841099fc7fa194c18575a48e7f69ba7839f4905555500b9b69" gracePeriod=30 Oct 30 22:04:02.157973 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:02.157929 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:03.451641 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:03.451609 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-krc6t"] Oct 30 22:04:03.456062 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:03.456032 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-krc6t"] Oct 30 22:04:03.476686 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:03.476656 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vt7gs"] Oct 30 22:04:03.477079 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:03.477064 5801 cpu_manager.go:414] "containerMap: removing container" podUID="b67bf626-8615-46e2-8e35-7bb7009fc11b" containerName="revoke-keys" Oct 30 22:04:03.477153 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:03.477080 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b67bf626-8615-46e2-8e35-7bb7009fc11b" containerName="revoke-keys" Oct 30 22:04:03.477209 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:03.477153 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="b67bf626-8615-46e2-8e35-7bb7009fc11b" containerName="revoke-keys" Oct 30 22:04:03.481453 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:03.481434 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vt7gs" Oct 30 22:04:03.487736 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:03.487709 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:04:03.489295 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:03.488920 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:04:03.489295 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:03.488936 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:04:03.489295 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:03.488944 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 22:04:03.490307 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:03.490286 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vt7gs"] Oct 30 22:04:03.522644 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:03.522621 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bdbaa88-f859-4e2a-8cee-8326a6b2fba9" path="/var/lib/kubelet/pods/1bdbaa88-f859-4e2a-8cee-8326a6b2fba9/volumes" Oct 30 22:04:03.628672 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:03.628647 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d90664e3-47d3-4247-8804-10f1f7d3c09d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vt7gs\" (UID: \"d90664e3-47d3-4247-8804-10f1f7d3c09d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vt7gs" Oct 30 22:04:03.628803 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:03.628701 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d90664e3-47d3-4247-8804-10f1f7d3c09d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vt7gs\" (UID: \"d90664e3-47d3-4247-8804-10f1f7d3c09d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vt7gs" Oct 30 22:04:03.628866 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:03.628828 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlmpw\" (UniqueName: \"kubernetes.io/projected/d90664e3-47d3-4247-8804-10f1f7d3c09d-kube-api-access-xlmpw\") pod \"maas-api-revoke-sub-3d4156f1-vt7gs\" (UID: \"d90664e3-47d3-4247-8804-10f1f7d3c09d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vt7gs" Oct 30 22:04:03.730026 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:03.729971 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d90664e3-47d3-4247-8804-10f1f7d3c09d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vt7gs\" (UID: \"d90664e3-47d3-4247-8804-10f1f7d3c09d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vt7gs" Oct 30 22:04:03.730117 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:03.730026 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d90664e3-47d3-4247-8804-10f1f7d3c09d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vt7gs\" (UID: \"d90664e3-47d3-4247-8804-10f1f7d3c09d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vt7gs" Oct 30 22:04:03.730117 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:03.730071 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlmpw\" (UniqueName: \"kubernetes.io/projected/d90664e3-47d3-4247-8804-10f1f7d3c09d-kube-api-access-xlmpw\") pod \"maas-api-revoke-sub-3d4156f1-vt7gs\" (UID: \"d90664e3-47d3-4247-8804-10f1f7d3c09d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vt7gs" Oct 30 22:04:03.730661 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:03.730642 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d90664e3-47d3-4247-8804-10f1f7d3c09d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vt7gs\" (UID: \"d90664e3-47d3-4247-8804-10f1f7d3c09d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vt7gs" Oct 30 22:04:03.732398 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:03.732375 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d90664e3-47d3-4247-8804-10f1f7d3c09d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vt7gs\" (UID: \"d90664e3-47d3-4247-8804-10f1f7d3c09d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vt7gs" Oct 30 22:04:03.742369 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:03.742345 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlmpw\" (UniqueName: \"kubernetes.io/projected/d90664e3-47d3-4247-8804-10f1f7d3c09d-kube-api-access-xlmpw\") pod \"maas-api-revoke-sub-3d4156f1-vt7gs\" (UID: \"d90664e3-47d3-4247-8804-10f1f7d3c09d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vt7gs" Oct 30 22:04:03.795293 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:03.795273 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vt7gs" Oct 30 22:04:03.967975 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:03.967939 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" podUID="a755d2ec-37af-488d-89ea-a4c0d0836bcd" containerName="main" containerID="cri-o://9477589d9e16a36fd3b39f72f73f86868334e52a8d1e43faa3dff424e9dd0b7f" gracePeriod=45 Oct 30 22:04:04.071280 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:04.071251 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vt7gs"] Oct 30 22:04:04.073362 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:04:04.073336 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90664e3_47d3_4247_8804_10f1f7d3c09d.slice/crio-89e5f93250321f11fcf33363123b4fa3b83223ebaa8f0dd4b5e32d68064c1162 WatchSource:0}: Error finding container 89e5f93250321f11fcf33363123b4fa3b83223ebaa8f0dd4b5e32d68064c1162: Status 404 returned error can't find the container with id 89e5f93250321f11fcf33363123b4fa3b83223ebaa8f0dd4b5e32d68064c1162 Oct 30 22:04:04.100328 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:04.100301 5801 generic.go:356] "Generic (PLEG): container finished" podID="a755d2ec-37af-488d-89ea-a4c0d0836bcd" containerID="9477589d9e16a36fd3b39f72f73f86868334e52a8d1e43faa3dff424e9dd0b7f" exitCode=0 Oct 30 22:04:04.100437 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:04.100370 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" event={"ID":"a755d2ec-37af-488d-89ea-a4c0d0836bcd","Type":"ContainerDied","Data":"9477589d9e16a36fd3b39f72f73f86868334e52a8d1e43faa3dff424e9dd0b7f"} Oct 30 22:04:04.101895 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:04.101872 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vt7gs" event={"ID":"d90664e3-47d3-4247-8804-10f1f7d3c09d","Type":"ContainerStarted","Data":"89e5f93250321f11fcf33363123b4fa3b83223ebaa8f0dd4b5e32d68064c1162"} Oct 30 22:04:04.226394 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:04.226373 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:04:04.336164 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:04.336136 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-model-cache\") pod \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\" (UID: \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\") " Oct 30 22:04:04.336329 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:04.336170 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-home\" (UniqueName: \"kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-home\") pod \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\" (UID: \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\") " Oct 30 22:04:04.336329 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:04.336191 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-dshm\") pod \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\" (UID: \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\") " Oct 30 22:04:04.336329 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:04.336238 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a755d2ec-37af-488d-89ea-a4c0d0836bcd-tls-certs\" (UniqueName: \"kubernetes.io/secret/a755d2ec-37af-488d-89ea-a4c0d0836bcd-tls-certs\") pod \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\" (UID: \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\") " Oct 30 22:04:04.336329 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:04.336274 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a755d2ec-37af-488d-89ea-a4c0d0836bcd-kube-api-access-bbjl2\" (UniqueName: \"kubernetes.io/projected/a755d2ec-37af-488d-89ea-a4c0d0836bcd-kube-api-access-bbjl2\") pod \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\" (UID: \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\") " Oct 30 22:04:04.336573 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:04.336399 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-tmp-dir\") pod \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\" (UID: \"a755d2ec-37af-488d-89ea-a4c0d0836bcd\") " Oct 30 22:04:04.336573 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:04.336453 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-model-cache" pod "a755d2ec-37af-488d-89ea-a4c0d0836bcd" (UID: "a755d2ec-37af-488d-89ea-a4c0d0836bcd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:04.336573 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:04.336459 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-home" pod "a755d2ec-37af-488d-89ea-a4c0d0836bcd" (UID: "a755d2ec-37af-488d-89ea-a4c0d0836bcd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:04.336709 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:04.336643 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-tmp-dir" pod "a755d2ec-37af-488d-89ea-a4c0d0836bcd" (UID: "a755d2ec-37af-488d-89ea-a4c0d0836bcd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:04.336817 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:04.336796 5801 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:04.336859 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:04.336823 5801 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:04.336859 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:04.336838 5801 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:04.353881 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:04.353815 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a755d2ec-37af-488d-89ea-a4c0d0836bcd-kube-api-access-bbjl2" pod "a755d2ec-37af-488d-89ea-a4c0d0836bcd" (UID: "a755d2ec-37af-488d-89ea-a4c0d0836bcd"). InnerVolumeSpecName "kube-api-access-bbjl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:04.354139 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:04.353866 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a755d2ec-37af-488d-89ea-a4c0d0836bcd-tls-certs" pod "a755d2ec-37af-488d-89ea-a4c0d0836bcd" (UID: "a755d2ec-37af-488d-89ea-a4c0d0836bcd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:04.354139 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:04.354094 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-dshm" pod "a755d2ec-37af-488d-89ea-a4c0d0836bcd" (UID: "a755d2ec-37af-488d-89ea-a4c0d0836bcd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:04.437550 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:04.437527 5801 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a755d2ec-37af-488d-89ea-a4c0d0836bcd-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:04.437550 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:04.437547 5801 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a755d2ec-37af-488d-89ea-a4c0d0836bcd-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:04.437693 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:04.437558 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbjl2\" (UniqueName: \"kubernetes.io/projected/a755d2ec-37af-488d-89ea-a4c0d0836bcd-kube-api-access-bbjl2\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:05.108612 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:05.108574 5801 generic.go:356] "Generic (PLEG): container finished" podID="d90664e3-47d3-4247-8804-10f1f7d3c09d" containerID="f16f9bd53e577c6a405956985bb96c061ca424eaa387852c79291ecd03bdb4e0" exitCode=0 Oct 30 22:04:05.109060 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:05.108652 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vt7gs" event={"ID":"d90664e3-47d3-4247-8804-10f1f7d3c09d","Type":"ContainerDied","Data":"f16f9bd53e577c6a405956985bb96c061ca424eaa387852c79291ecd03bdb4e0"} Oct 30 22:04:05.110021 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:05.110000 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" Oct 30 22:04:05.110147 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:05.110003 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s" event={"ID":"a755d2ec-37af-488d-89ea-a4c0d0836bcd","Type":"ContainerDied","Data":"271cd5367487eec379ecff5dc960d08fb3e7a3bf36f447ad7ad6ba1a72920b95"} Oct 30 22:04:05.110147 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:05.110130 5801 scope.go:122] "RemoveContainer" containerID="9477589d9e16a36fd3b39f72f73f86868334e52a8d1e43faa3dff424e9dd0b7f" Oct 30 22:04:05.142876 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:05.142853 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s"] Oct 30 22:04:05.145737 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:05.145716 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-fa15d4/simulator-single-main-fa15d4-kserve-6775cbccb9-m2v4s"] Oct 30 22:04:05.522366 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:05.522294 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a755d2ec-37af-488d-89ea-a4c0d0836bcd" path="/var/lib/kubelet/pods/a755d2ec-37af-488d-89ea-a4c0d0836bcd/volumes" Oct 30 22:04:06.250715 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:06.250693 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vt7gs" Oct 30 22:04:06.360527 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:06.360499 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d90664e3-47d3-4247-8804-10f1f7d3c09d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d90664e3-47d3-4247-8804-10f1f7d3c09d-maas-api-openshift-ca\") pod \"d90664e3-47d3-4247-8804-10f1f7d3c09d\" (UID: \"d90664e3-47d3-4247-8804-10f1f7d3c09d\") " Oct 30 22:04:06.360702 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:06.360643 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d90664e3-47d3-4247-8804-10f1f7d3c09d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d90664e3-47d3-4247-8804-10f1f7d3c09d-maas-api-xks-ca\") pod \"d90664e3-47d3-4247-8804-10f1f7d3c09d\" (UID: \"d90664e3-47d3-4247-8804-10f1f7d3c09d\") " Oct 30 22:04:06.360702 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:06.360673 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d90664e3-47d3-4247-8804-10f1f7d3c09d-kube-api-access-xlmpw\" (UniqueName: \"kubernetes.io/projected/d90664e3-47d3-4247-8804-10f1f7d3c09d-kube-api-access-xlmpw\") pod \"d90664e3-47d3-4247-8804-10f1f7d3c09d\" (UID: \"d90664e3-47d3-4247-8804-10f1f7d3c09d\") " Oct 30 22:04:06.360892 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:06.360857 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d90664e3-47d3-4247-8804-10f1f7d3c09d-maas-api-openshift-ca" pod "d90664e3-47d3-4247-8804-10f1f7d3c09d" (UID: "d90664e3-47d3-4247-8804-10f1f7d3c09d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:06.362797 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:06.362770 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d90664e3-47d3-4247-8804-10f1f7d3c09d-maas-api-xks-ca" pod "d90664e3-47d3-4247-8804-10f1f7d3c09d" (UID: "d90664e3-47d3-4247-8804-10f1f7d3c09d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:06.362920 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:06.362895 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90664e3-47d3-4247-8804-10f1f7d3c09d-kube-api-access-xlmpw" pod "d90664e3-47d3-4247-8804-10f1f7d3c09d" (UID: "d90664e3-47d3-4247-8804-10f1f7d3c09d"). InnerVolumeSpecName "kube-api-access-xlmpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:06.461703 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:06.461674 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d90664e3-47d3-4247-8804-10f1f7d3c09d-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:06.461703 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:06.461699 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d90664e3-47d3-4247-8804-10f1f7d3c09d-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:06.461864 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:06.461710 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlmpw\" (UniqueName: \"kubernetes.io/projected/d90664e3-47d3-4247-8804-10f1f7d3c09d-kube-api-access-xlmpw\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:07.123108 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.123080 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vt7gs" Oct 30 22:04:07.123290 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.123106 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vt7gs" event={"ID":"d90664e3-47d3-4247-8804-10f1f7d3c09d","Type":"ContainerDied","Data":"89e5f93250321f11fcf33363123b4fa3b83223ebaa8f0dd4b5e32d68064c1162"} Oct 30 22:04:07.123290 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.123151 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89e5f93250321f11fcf33363123b4fa3b83223ebaa8f0dd4b5e32d68064c1162" Oct 30 22:04:07.316213 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.316174 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh" podUID="7b7f8a19-123d-4368-8b73-10361bc06a26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:04:07.316665 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.316351 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz" podUID="21a1ae2e-9dfb-48c7-829f-ee129d480d56" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:04:07.355665 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.355642 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:04:07.469580 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.469493 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-credential-socket\") pod \"258e5cf7-89af-4f93-8423-24e4897e66e6\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " Oct 30 22:04:07.469580 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.469559 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-workload-certs\") pod \"258e5cf7-89af-4f93-8423-24e4897e66e6\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " Oct 30 22:04:07.469854 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.469591 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-data\") pod \"258e5cf7-89af-4f93-8423-24e4897e66e6\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " Oct 30 22:04:07.469854 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.469620 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/258e5cf7-89af-4f93-8423-24e4897e66e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/258e5cf7-89af-4f93-8423-24e4897e66e6-istiod-ca-cert\") pod \"258e5cf7-89af-4f93-8423-24e4897e66e6\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " Oct 30 22:04:07.469854 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.469655 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-workload-socket\") pod \"258e5cf7-89af-4f93-8423-24e4897e66e6\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " Oct 30 22:04:07.469854 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.469699 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-token\" (UniqueName: \"kubernetes.io/projected/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-token\") pod \"258e5cf7-89af-4f93-8423-24e4897e66e6\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " Oct 30 22:04:07.469854 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.469807 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-credential-socket" pod "258e5cf7-89af-4f93-8423-24e4897e66e6" (UID: "258e5cf7-89af-4f93-8423-24e4897e66e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:07.469854 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.469818 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-envoy\") pod \"258e5cf7-89af-4f93-8423-24e4897e66e6\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " Oct 30 22:04:07.470166 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.469892 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-podinfo\") pod \"258e5cf7-89af-4f93-8423-24e4897e66e6\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " Oct 30 22:04:07.470166 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.469916 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/258e5cf7-89af-4f93-8423-24e4897e66e6-kube-api-access-cj5dd\" (UniqueName: \"kubernetes.io/projected/258e5cf7-89af-4f93-8423-24e4897e66e6-kube-api-access-cj5dd\") pod \"258e5cf7-89af-4f93-8423-24e4897e66e6\" (UID: \"258e5cf7-89af-4f93-8423-24e4897e66e6\") " Oct 30 22:04:07.470166 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.469885 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-workload-certs" pod "258e5cf7-89af-4f93-8423-24e4897e66e6" (UID: "258e5cf7-89af-4f93-8423-24e4897e66e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:07.470166 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.469955 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-data" pod "258e5cf7-89af-4f93-8423-24e4897e66e6" (UID: "258e5cf7-89af-4f93-8423-24e4897e66e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:07.470166 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.469972 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-workload-socket" pod "258e5cf7-89af-4f93-8423-24e4897e66e6" (UID: "258e5cf7-89af-4f93-8423-24e4897e66e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:07.470439 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.470280 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:07.470439 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.470300 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:07.470439 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.470364 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:07.470439 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.470377 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:07.470439 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.470321 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/258e5cf7-89af-4f93-8423-24e4897e66e6-istiod-ca-cert" pod "258e5cf7-89af-4f93-8423-24e4897e66e6" (UID: "258e5cf7-89af-4f93-8423-24e4897e66e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:07.472203 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.472174 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-podinfo" pod "258e5cf7-89af-4f93-8423-24e4897e66e6" (UID: "258e5cf7-89af-4f93-8423-24e4897e66e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:04:07.472349 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.472180 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-token" pod "258e5cf7-89af-4f93-8423-24e4897e66e6" (UID: "258e5cf7-89af-4f93-8423-24e4897e66e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:07.472424 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.472406 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-envoy" pod "258e5cf7-89af-4f93-8423-24e4897e66e6" (UID: "258e5cf7-89af-4f93-8423-24e4897e66e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:07.472509 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.472495 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/258e5cf7-89af-4f93-8423-24e4897e66e6-kube-api-access-cj5dd" pod "258e5cf7-89af-4f93-8423-24e4897e66e6" (UID: "258e5cf7-89af-4f93-8423-24e4897e66e6"). InnerVolumeSpecName "kube-api-access-cj5dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:07.571742 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.571715 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:07.571742 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.571741 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:07.571913 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.571775 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/258e5cf7-89af-4f93-8423-24e4897e66e6-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:07.571913 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.571787 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj5dd\" (UniqueName: \"kubernetes.io/projected/258e5cf7-89af-4f93-8423-24e4897e66e6-kube-api-access-cj5dd\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:07.571913 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:07.571796 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/258e5cf7-89af-4f93-8423-24e4897e66e6-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:08.129168 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:08.129127 5801 generic.go:356] "Generic (PLEG): container finished" podID="258e5cf7-89af-4f93-8423-24e4897e66e6" containerID="9b5b87047fb640841099fc7fa194c18575a48e7f69ba7839f4905555500b9b69" exitCode=0 Oct 30 22:04:08.129356 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:08.129180 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" Oct 30 22:04:08.129356 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:08.129211 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" event={"ID":"258e5cf7-89af-4f93-8423-24e4897e66e6","Type":"ContainerDied","Data":"9b5b87047fb640841099fc7fa194c18575a48e7f69ba7839f4905555500b9b69"} Oct 30 22:04:08.129356 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:08.129266 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d" event={"ID":"258e5cf7-89af-4f93-8423-24e4897e66e6","Type":"ContainerDied","Data":"d7ed780618dfb84679c9098205db189d9ceb6051047acbe358c9b28535feaaba"} Oct 30 22:04:08.129356 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:08.129291 5801 scope.go:122] "RemoveContainer" containerID="9b5b87047fb640841099fc7fa194c18575a48e7f69ba7839f4905555500b9b69" Oct 30 22:04:08.138519 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:08.138488 5801 scope.go:122] "RemoveContainer" containerID="5d5a06a288aa7ae080b81d52697937159439d93dd1d485deb72c833e3bf0fa1b" Oct 30 22:04:08.146902 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:08.146691 5801 scope.go:122] "RemoveContainer" containerID="9b5b87047fb640841099fc7fa194c18575a48e7f69ba7839f4905555500b9b69" Oct 30 22:04:08.147219 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:04:08.147047 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b5b87047fb640841099fc7fa194c18575a48e7f69ba7839f4905555500b9b69\": container with ID starting with 9b5b87047fb640841099fc7fa194c18575a48e7f69ba7839f4905555500b9b69 not found: ID does not exist" containerID="9b5b87047fb640841099fc7fa194c18575a48e7f69ba7839f4905555500b9b69" Oct 30 22:04:08.147219 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:08.147079 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b5b87047fb640841099fc7fa194c18575a48e7f69ba7839f4905555500b9b69"} err="failed to get container status \"9b5b87047fb640841099fc7fa194c18575a48e7f69ba7839f4905555500b9b69\": rpc error: code = NotFound desc = could not find container \"9b5b87047fb640841099fc7fa194c18575a48e7f69ba7839f4905555500b9b69\": container with ID starting with 9b5b87047fb640841099fc7fa194c18575a48e7f69ba7839f4905555500b9b69 not found: ID does not exist" Oct 30 22:04:08.147219 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:08.147119 5801 scope.go:122] "RemoveContainer" containerID="5d5a06a288aa7ae080b81d52697937159439d93dd1d485deb72c833e3bf0fa1b" Oct 30 22:04:08.147412 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:04:08.147382 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d5a06a288aa7ae080b81d52697937159439d93dd1d485deb72c833e3bf0fa1b\": container with ID starting with 5d5a06a288aa7ae080b81d52697937159439d93dd1d485deb72c833e3bf0fa1b not found: ID does not exist" containerID="5d5a06a288aa7ae080b81d52697937159439d93dd1d485deb72c833e3bf0fa1b" Oct 30 22:04:08.147457 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:08.147409 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d5a06a288aa7ae080b81d52697937159439d93dd1d485deb72c833e3bf0fa1b"} err="failed to get container status \"5d5a06a288aa7ae080b81d52697937159439d93dd1d485deb72c833e3bf0fa1b\": rpc error: code = NotFound desc = could not find container \"5d5a06a288aa7ae080b81d52697937159439d93dd1d485deb72c833e3bf0fa1b\": container with ID starting with 5d5a06a288aa7ae080b81d52697937159439d93dd1d485deb72c833e3bf0fa1b not found: ID does not exist" Oct 30 22:04:08.149224 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:08.149209 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d"] Oct 30 22:04:08.153331 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:08.153311 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fa15d4-openshift-default-6775d7645-d7r9d"] Oct 30 22:04:09.522479 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:09.522445 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="258e5cf7-89af-4f93-8423-24e4897e66e6" path="/var/lib/kubelet/pods/258e5cf7-89af-4f93-8423-24e4897e66e6/volumes" Oct 30 22:04:16.751379 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:16.750864 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:16.751379 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:16.750951 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" podUID="fda6dedb-779c-49d8-8fcf-ba31f73402ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:17.323701 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:17.323262 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh" podUID="7b7f8a19-123d-4368-8b73-10361bc06a26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:04:17.324290 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:17.324255 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz" podUID="21a1ae2e-9dfb-48c7-829f-ee129d480d56" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:04:20.045451 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.039103 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh" Oct 30 22:04:20.049727 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.049701 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz" Oct 30 22:04:20.150713 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.150680 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b7f8a19-123d-4368-8b73-10361bc06a26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b7f8a19-123d-4368-8b73-10361bc06a26-plugins-config-volume\") pod \"7b7f8a19-123d-4368-8b73-10361bc06a26\" (UID: \"7b7f8a19-123d-4368-8b73-10361bc06a26\") " Oct 30 22:04:20.150926 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.150726 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21a1ae2e-9dfb-48c7-829f-ee129d480d56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a1ae2e-9dfb-48c7-829f-ee129d480d56-plugins-config-volume\") pod \"21a1ae2e-9dfb-48c7-829f-ee129d480d56\" (UID: \"21a1ae2e-9dfb-48c7-829f-ee129d480d56\") " Oct 30 22:04:20.150926 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.150767 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a1ae2e-9dfb-48c7-829f-ee129d480d56-kube-api-access-jwsdx\" (UniqueName: \"kubernetes.io/projected/21a1ae2e-9dfb-48c7-829f-ee129d480d56-kube-api-access-jwsdx\") pod \"21a1ae2e-9dfb-48c7-829f-ee129d480d56\" (UID: \"21a1ae2e-9dfb-48c7-829f-ee129d480d56\") " Oct 30 22:04:20.150926 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.150795 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b7f8a19-123d-4368-8b73-10361bc06a26-kube-api-access-vqbdl\" (UniqueName: \"kubernetes.io/projected/7b7f8a19-123d-4368-8b73-10361bc06a26-kube-api-access-vqbdl\") pod \"7b7f8a19-123d-4368-8b73-10361bc06a26\" (UID: \"7b7f8a19-123d-4368-8b73-10361bc06a26\") " Oct 30 22:04:20.151124 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.151034 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b7f8a19-123d-4368-8b73-10361bc06a26-plugins-config-volume" pod "7b7f8a19-123d-4368-8b73-10361bc06a26" (UID: "7b7f8a19-123d-4368-8b73-10361bc06a26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:20.151215 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.151185 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a1ae2e-9dfb-48c7-829f-ee129d480d56-plugins-config-volume" pod "21a1ae2e-9dfb-48c7-829f-ee129d480d56" (UID: "21a1ae2e-9dfb-48c7-829f-ee129d480d56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:20.166020 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.165981 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a1ae2e-9dfb-48c7-829f-ee129d480d56-kube-api-access-jwsdx" pod "21a1ae2e-9dfb-48c7-829f-ee129d480d56" (UID: "21a1ae2e-9dfb-48c7-829f-ee129d480d56"). InnerVolumeSpecName "kube-api-access-jwsdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:20.179778 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.175963 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b7f8a19-123d-4368-8b73-10361bc06a26-kube-api-access-vqbdl" pod "7b7f8a19-123d-4368-8b73-10361bc06a26" (UID: "7b7f8a19-123d-4368-8b73-10361bc06a26"). InnerVolumeSpecName "kube-api-access-vqbdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:20.240068 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.240029 5801 generic.go:356] "Generic (PLEG): container finished" podID="21a1ae2e-9dfb-48c7-829f-ee129d480d56" containerID="3fc90682231086322655f7f37b439464b0e6812da033ff2ba624aba8c3bc037d" exitCode=0 Oct 30 22:04:20.240285 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.240077 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz" Oct 30 22:04:20.240285 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.240105 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz" event={"ID":"21a1ae2e-9dfb-48c7-829f-ee129d480d56","Type":"ContainerDied","Data":"3fc90682231086322655f7f37b439464b0e6812da033ff2ba624aba8c3bc037d"} Oct 30 22:04:20.240285 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.240188 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz" event={"ID":"21a1ae2e-9dfb-48c7-829f-ee129d480d56","Type":"ContainerDied","Data":"1f748a5f668b868ee73d2976a75946593895e67974d9ce6b773268afac3c2743"} Oct 30 22:04:20.240285 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.240213 5801 scope.go:122] "RemoveContainer" containerID="3fc90682231086322655f7f37b439464b0e6812da033ff2ba624aba8c3bc037d" Oct 30 22:04:20.242623 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.242596 5801 generic.go:356] "Generic (PLEG): container finished" podID="7b7f8a19-123d-4368-8b73-10361bc06a26" containerID="8d2127dd3644a8797fd48de09470101869a8cea901ff8c406d7f267be87e2710" exitCode=0 Oct 30 22:04:20.242737 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.242708 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh" event={"ID":"7b7f8a19-123d-4368-8b73-10361bc06a26","Type":"ContainerDied","Data":"8d2127dd3644a8797fd48de09470101869a8cea901ff8c406d7f267be87e2710"} Oct 30 22:04:20.242825 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.242768 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh" event={"ID":"7b7f8a19-123d-4368-8b73-10361bc06a26","Type":"ContainerDied","Data":"e039ec92aeb95809e3a099f4f02260b66ca4168e11ee9948736f7596c68b981b"} Oct 30 22:04:20.242884 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.242821 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh" Oct 30 22:04:20.251461 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.251439 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b7f8a19-123d-4368-8b73-10361bc06a26-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.251566 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.251464 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a1ae2e-9dfb-48c7-829f-ee129d480d56-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.251566 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.251482 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwsdx\" (UniqueName: \"kubernetes.io/projected/21a1ae2e-9dfb-48c7-829f-ee129d480d56-kube-api-access-jwsdx\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.251566 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.251491 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqbdl\" (UniqueName: \"kubernetes.io/projected/7b7f8a19-123d-4368-8b73-10361bc06a26-kube-api-access-vqbdl\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:20.280506 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.265432 5801 scope.go:122] "RemoveContainer" containerID="3fc90682231086322655f7f37b439464b0e6812da033ff2ba624aba8c3bc037d" Oct 30 22:04:20.299311 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.299246 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz"] Oct 30 22:04:20.328340 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:04:20.328099 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fc90682231086322655f7f37b439464b0e6812da033ff2ba624aba8c3bc037d\": container with ID starting with 3fc90682231086322655f7f37b439464b0e6812da033ff2ba624aba8c3bc037d not found: ID does not exist" containerID="3fc90682231086322655f7f37b439464b0e6812da033ff2ba624aba8c3bc037d" Oct 30 22:04:20.328340 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.328155 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fc90682231086322655f7f37b439464b0e6812da033ff2ba624aba8c3bc037d"} err="failed to get container status \"3fc90682231086322655f7f37b439464b0e6812da033ff2ba624aba8c3bc037d\": rpc error: code = NotFound desc = could not find container \"3fc90682231086322655f7f37b439464b0e6812da033ff2ba624aba8c3bc037d\": container with ID starting with 3fc90682231086322655f7f37b439464b0e6812da033ff2ba624aba8c3bc037d not found: ID does not exist" Oct 30 22:04:20.328340 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.328184 5801 scope.go:122] "RemoveContainer" containerID="8d2127dd3644a8797fd48de09470101869a8cea901ff8c406d7f267be87e2710" Oct 30 22:04:20.338925 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.337644 5801 scope.go:122] "RemoveContainer" containerID="8d2127dd3644a8797fd48de09470101869a8cea901ff8c406d7f267be87e2710" Oct 30 22:04:20.338925 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:04:20.337982 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d2127dd3644a8797fd48de09470101869a8cea901ff8c406d7f267be87e2710\": container with ID starting with 8d2127dd3644a8797fd48de09470101869a8cea901ff8c406d7f267be87e2710 not found: ID does not exist" containerID="8d2127dd3644a8797fd48de09470101869a8cea901ff8c406d7f267be87e2710" Oct 30 22:04:20.338925 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.338011 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d2127dd3644a8797fd48de09470101869a8cea901ff8c406d7f267be87e2710"} err="failed to get container status \"8d2127dd3644a8797fd48de09470101869a8cea901ff8c406d7f267be87e2710\": rpc error: code = NotFound desc = could not find container \"8d2127dd3644a8797fd48de09470101869a8cea901ff8c406d7f267be87e2710\": container with ID starting with 8d2127dd3644a8797fd48de09470101869a8cea901ff8c406d7f267be87e2710 not found: ID does not exist" Oct 30 22:04:20.339219 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.339199 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fa15d4-6fb76jjbmz"] Oct 30 22:04:20.341498 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.341477 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh"] Oct 30 22:04:20.343018 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:20.342994 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fa15d4-8477bb7d9m98lh"] Oct 30 22:04:21.522891 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:21.522859 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21a1ae2e-9dfb-48c7-829f-ee129d480d56" path="/var/lib/kubelet/pods/21a1ae2e-9dfb-48c7-829f-ee129d480d56/volumes" Oct 30 22:04:21.523293 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:21.523279 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b7f8a19-123d-4368-8b73-10361bc06a26" path="/var/lib/kubelet/pods/7b7f8a19-123d-4368-8b73-10361bc06a26/volumes" Oct 30 22:04:25.727285 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:25.727245 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:27.173362 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:27.173324 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:30.277474 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:04:30.277432 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:04:33.226903 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:33.226864 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:58.934971 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:04:58.934933 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:00.278943 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:05:00.278902 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:05:30.278164 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:05:30.278079 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:05:46.758897 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:46.758842 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:46.759603 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:46.759502 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7wxsr" podUID="fda6dedb-779c-49d8-8fcf-ba31f73402ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:48.180678 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.180647 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d5rc9"] Oct 30 22:05:48.181098 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.181082 5801 cpu_manager.go:414] "containerMap: removing container" podUID="d90664e3-47d3-4247-8804-10f1f7d3c09d" containerName="revoke-keys" Oct 30 22:05:48.181148 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.181099 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d90664e3-47d3-4247-8804-10f1f7d3c09d" containerName="revoke-keys" Oct 30 22:05:48.181148 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.181110 5801 cpu_manager.go:414] "containerMap: removing container" podUID="258e5cf7-89af-4f93-8423-24e4897e66e6" containerName="istio-proxy" Oct 30 22:05:48.181148 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.181115 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="258e5cf7-89af-4f93-8423-24e4897e66e6" containerName="istio-proxy" Oct 30 22:05:48.181148 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.181122 5801 cpu_manager.go:414] "containerMap: removing container" podUID="a755d2ec-37af-488d-89ea-a4c0d0836bcd" containerName="main" Oct 30 22:05:48.181148 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.181127 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a755d2ec-37af-488d-89ea-a4c0d0836bcd" containerName="main" Oct 30 22:05:48.181148 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.181143 5801 cpu_manager.go:414] "containerMap: removing container" podUID="21a1ae2e-9dfb-48c7-829f-ee129d480d56" containerName="payload-pre-processing" Oct 30 22:05:48.181148 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.181148 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21a1ae2e-9dfb-48c7-829f-ee129d480d56" containerName="payload-pre-processing" Oct 30 22:05:48.181378 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.181157 5801 cpu_manager.go:414] "containerMap: removing container" podUID="7b7f8a19-123d-4368-8b73-10361bc06a26" containerName="payload-processing" Oct 30 22:05:48.181378 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.181162 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b7f8a19-123d-4368-8b73-10361bc06a26" containerName="payload-processing" Oct 30 22:05:48.181378 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.181225 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="d90664e3-47d3-4247-8804-10f1f7d3c09d" containerName="revoke-keys" Oct 30 22:05:48.181378 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.181233 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="258e5cf7-89af-4f93-8423-24e4897e66e6" containerName="istio-proxy" Oct 30 22:05:48.181378 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.181239 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="a755d2ec-37af-488d-89ea-a4c0d0836bcd" containerName="main" Oct 30 22:05:48.181378 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.181247 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="21a1ae2e-9dfb-48c7-829f-ee129d480d56" containerName="payload-pre-processing" Oct 30 22:05:48.181378 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.181252 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b7f8a19-123d-4368-8b73-10361bc06a26" containerName="payload-processing" Oct 30 22:05:48.181378 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.181259 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="258e5cf7-89af-4f93-8423-24e4897e66e6" containerName="istio-proxy" Oct 30 22:05:48.184208 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.184191 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d5rc9" Oct 30 22:05:48.191861 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.191837 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:05:48.192469 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.192445 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:05:48.193431 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.193397 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 22:05:48.194614 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.194592 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:05:48.195959 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.195938 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d5rc9"] Oct 30 22:05:48.217141 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.217118 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/620fbb96-a894-4901-89fc-55a0950a0c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-d5rc9\" (UID: \"620fbb96-a894-4901-89fc-55a0950a0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d5rc9" Oct 30 22:05:48.217268 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.217160 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/620fbb96-a894-4901-89fc-55a0950a0c8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-d5rc9\" (UID: \"620fbb96-a894-4901-89fc-55a0950a0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d5rc9" Oct 30 22:05:48.217268 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.217189 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d522s\" (UniqueName: \"kubernetes.io/projected/620fbb96-a894-4901-89fc-55a0950a0c8d-kube-api-access-d522s\") pod \"maas-api-revoke-sub-d576e68b-d5rc9\" (UID: \"620fbb96-a894-4901-89fc-55a0950a0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d5rc9" Oct 30 22:05:48.318267 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.318232 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/620fbb96-a894-4901-89fc-55a0950a0c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-d5rc9\" (UID: \"620fbb96-a894-4901-89fc-55a0950a0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d5rc9" Oct 30 22:05:48.318424 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.318308 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/620fbb96-a894-4901-89fc-55a0950a0c8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-d5rc9\" (UID: \"620fbb96-a894-4901-89fc-55a0950a0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d5rc9" Oct 30 22:05:48.318424 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.318354 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d522s\" (UniqueName: \"kubernetes.io/projected/620fbb96-a894-4901-89fc-55a0950a0c8d-kube-api-access-d522s\") pod \"maas-api-revoke-sub-d576e68b-d5rc9\" (UID: \"620fbb96-a894-4901-89fc-55a0950a0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d5rc9" Oct 30 22:05:48.318977 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.318952 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/620fbb96-a894-4901-89fc-55a0950a0c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-d5rc9\" (UID: \"620fbb96-a894-4901-89fc-55a0950a0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d5rc9" Oct 30 22:05:48.320782 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.320742 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/620fbb96-a894-4901-89fc-55a0950a0c8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-d5rc9\" (UID: \"620fbb96-a894-4901-89fc-55a0950a0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d5rc9" Oct 30 22:05:48.331140 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.331118 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d522s\" (UniqueName: \"kubernetes.io/projected/620fbb96-a894-4901-89fc-55a0950a0c8d-kube-api-access-d522s\") pod \"maas-api-revoke-sub-d576e68b-d5rc9\" (UID: \"620fbb96-a894-4901-89fc-55a0950a0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d5rc9" Oct 30 22:05:48.498933 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.498878 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d5rc9" Oct 30 22:05:48.774533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:48.774510 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d5rc9"] Oct 30 22:05:48.777049 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:05:48.777021 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620fbb96_a894_4901_89fc_55a0950a0c8d.slice/crio-f6fe731e326bd6912d8a5f22c2f9a8bd2fd4233d7371c86e143e5158a0f7b838 WatchSource:0}: Error finding container f6fe731e326bd6912d8a5f22c2f9a8bd2fd4233d7371c86e143e5158a0f7b838: Status 404 returned error can't find the container with id f6fe731e326bd6912d8a5f22c2f9a8bd2fd4233d7371c86e143e5158a0f7b838 Oct 30 22:05:49.731029 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:49.730996 5801 generic.go:356] "Generic (PLEG): container finished" podID="620fbb96-a894-4901-89fc-55a0950a0c8d" containerID="515444b3db68347541f6a85f1a4bdc88025a1514ac81dea7d8984feda874d066" exitCode=0 Oct 30 22:05:49.731410 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:49.731039 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d5rc9" event={"ID":"620fbb96-a894-4901-89fc-55a0950a0c8d","Type":"ContainerDied","Data":"515444b3db68347541f6a85f1a4bdc88025a1514ac81dea7d8984feda874d066"} Oct 30 22:05:49.731410 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:49.731117 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d5rc9" event={"ID":"620fbb96-a894-4901-89fc-55a0950a0c8d","Type":"ContainerStarted","Data":"f6fe731e326bd6912d8a5f22c2f9a8bd2fd4233d7371c86e143e5158a0f7b838"} Oct 30 22:05:50.864713 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:50.864689 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d5rc9" Oct 30 22:05:50.941459 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:50.941430 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/620fbb96-a894-4901-89fc-55a0950a0c8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/620fbb96-a894-4901-89fc-55a0950a0c8d-maas-api-xks-ca\") pod \"620fbb96-a894-4901-89fc-55a0950a0c8d\" (UID: \"620fbb96-a894-4901-89fc-55a0950a0c8d\") " Oct 30 22:05:50.941617 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:50.941473 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/620fbb96-a894-4901-89fc-55a0950a0c8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/620fbb96-a894-4901-89fc-55a0950a0c8d-maas-api-openshift-ca\") pod \"620fbb96-a894-4901-89fc-55a0950a0c8d\" (UID: \"620fbb96-a894-4901-89fc-55a0950a0c8d\") " Oct 30 22:05:50.941617 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:50.941514 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/620fbb96-a894-4901-89fc-55a0950a0c8d-kube-api-access-d522s\" (UniqueName: \"kubernetes.io/projected/620fbb96-a894-4901-89fc-55a0950a0c8d-kube-api-access-d522s\") pod \"620fbb96-a894-4901-89fc-55a0950a0c8d\" (UID: \"620fbb96-a894-4901-89fc-55a0950a0c8d\") " Oct 30 22:05:50.941898 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:50.941871 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/620fbb96-a894-4901-89fc-55a0950a0c8d-maas-api-openshift-ca" pod "620fbb96-a894-4901-89fc-55a0950a0c8d" (UID: "620fbb96-a894-4901-89fc-55a0950a0c8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:50.943495 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:50.943470 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/620fbb96-a894-4901-89fc-55a0950a0c8d-maas-api-xks-ca" pod "620fbb96-a894-4901-89fc-55a0950a0c8d" (UID: "620fbb96-a894-4901-89fc-55a0950a0c8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:50.943594 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:50.943538 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620fbb96-a894-4901-89fc-55a0950a0c8d-kube-api-access-d522s" pod "620fbb96-a894-4901-89fc-55a0950a0c8d" (UID: "620fbb96-a894-4901-89fc-55a0950a0c8d"). InnerVolumeSpecName "kube-api-access-d522s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:51.042438 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:51.042380 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/620fbb96-a894-4901-89fc-55a0950a0c8d-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:51.042438 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:51.042406 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/620fbb96-a894-4901-89fc-55a0950a0c8d-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:51.042438 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:51.042416 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d522s\" (UniqueName: \"kubernetes.io/projected/620fbb96-a894-4901-89fc-55a0950a0c8d-kube-api-access-d522s\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:51.741568 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:51.741538 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d5rc9" Oct 30 22:05:51.741726 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:51.741540 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d5rc9" event={"ID":"620fbb96-a894-4901-89fc-55a0950a0c8d","Type":"ContainerDied","Data":"f6fe731e326bd6912d8a5f22c2f9a8bd2fd4233d7371c86e143e5158a0f7b838"} Oct 30 22:05:51.741726 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:51.741662 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6fe731e326bd6912d8a5f22c2f9a8bd2fd4233d7371c86e143e5158a0f7b838" Oct 30 22:05:58.988613 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:05:58.988558 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:00.279601 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:06:00.279555 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:06:07.276228 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:07.276191 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kb4qt"] Oct 30 22:06:07.276737 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:07.276616 5801 cpu_manager.go:414] "containerMap: removing container" podUID="258e5cf7-89af-4f93-8423-24e4897e66e6" containerName="istio-proxy" Oct 30 22:06:07.276737 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:07.276631 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="258e5cf7-89af-4f93-8423-24e4897e66e6" containerName="istio-proxy" Oct 30 22:06:07.276737 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:07.276659 5801 cpu_manager.go:414] "containerMap: removing container" podUID="620fbb96-a894-4901-89fc-55a0950a0c8d" containerName="revoke-keys" Oct 30 22:06:07.276737 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:07.276664 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="620fbb96-a894-4901-89fc-55a0950a0c8d" containerName="revoke-keys" Oct 30 22:06:07.276737 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:07.276727 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="620fbb96-a894-4901-89fc-55a0950a0c8d" containerName="revoke-keys" Oct 30 22:06:07.280131 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:07.280107 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kb4qt" Oct 30 22:06:07.284340 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:07.284313 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:06:07.285279 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:07.285240 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:06:07.285413 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:07.285297 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:06:07.285548 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:07.285531 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 22:06:07.290859 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:07.290837 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kb4qt"] Oct 30 22:06:07.431088 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:07.431057 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03dcbb5f-dd7f-4ab6-ad37-4452802ea050-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-kb4qt\" (UID: \"03dcbb5f-dd7f-4ab6-ad37-4452802ea050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kb4qt" Oct 30 22:06:07.431088 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:07.431090 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03dcbb5f-dd7f-4ab6-ad37-4452802ea050-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-kb4qt\" (UID: \"03dcbb5f-dd7f-4ab6-ad37-4452802ea050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kb4qt" Oct 30 22:06:07.431303 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:07.431151 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5ddf\" (UniqueName: \"kubernetes.io/projected/03dcbb5f-dd7f-4ab6-ad37-4452802ea050-kube-api-access-g5ddf\") pod \"maas-api-revoke-sub-d2bc93bc-kb4qt\" (UID: \"03dcbb5f-dd7f-4ab6-ad37-4452802ea050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kb4qt" Oct 30 22:06:07.532191 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:07.532106 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03dcbb5f-dd7f-4ab6-ad37-4452802ea050-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-kb4qt\" (UID: \"03dcbb5f-dd7f-4ab6-ad37-4452802ea050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kb4qt" Oct 30 22:06:07.532191 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:07.532145 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03dcbb5f-dd7f-4ab6-ad37-4452802ea050-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-kb4qt\" (UID: \"03dcbb5f-dd7f-4ab6-ad37-4452802ea050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kb4qt" Oct 30 22:06:07.532416 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:07.532198 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5ddf\" (UniqueName: \"kubernetes.io/projected/03dcbb5f-dd7f-4ab6-ad37-4452802ea050-kube-api-access-g5ddf\") pod \"maas-api-revoke-sub-d2bc93bc-kb4qt\" (UID: \"03dcbb5f-dd7f-4ab6-ad37-4452802ea050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kb4qt" Oct 30 22:06:07.532801 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:07.532748 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03dcbb5f-dd7f-4ab6-ad37-4452802ea050-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-kb4qt\" (UID: \"03dcbb5f-dd7f-4ab6-ad37-4452802ea050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kb4qt" Oct 30 22:06:07.534514 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:07.534482 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03dcbb5f-dd7f-4ab6-ad37-4452802ea050-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-kb4qt\" (UID: \"03dcbb5f-dd7f-4ab6-ad37-4452802ea050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kb4qt" Oct 30 22:06:07.544820 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:07.544794 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5ddf\" (UniqueName: \"kubernetes.io/projected/03dcbb5f-dd7f-4ab6-ad37-4452802ea050-kube-api-access-g5ddf\") pod \"maas-api-revoke-sub-d2bc93bc-kb4qt\" (UID: \"03dcbb5f-dd7f-4ab6-ad37-4452802ea050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kb4qt" Oct 30 22:06:07.594136 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:07.594106 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kb4qt" Oct 30 22:06:07.872658 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:07.872634 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kb4qt"] Oct 30 22:06:07.875157 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:06:07.875132 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03dcbb5f_dd7f_4ab6_ad37_4452802ea050.slice/crio-d94c38a39571b9268d647364f0361fe0f083d93fb587015792a63e73ec2fadb8 WatchSource:0}: Error finding container d94c38a39571b9268d647364f0361fe0f083d93fb587015792a63e73ec2fadb8: Status 404 returned error can't find the container with id d94c38a39571b9268d647364f0361fe0f083d93fb587015792a63e73ec2fadb8 Oct 30 22:06:08.865786 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:08.865735 5801 generic.go:356] "Generic (PLEG): container finished" podID="03dcbb5f-dd7f-4ab6-ad37-4452802ea050" containerID="e5b15a6702c2cd8418a4ebf3d013ad37043bd7db7a66eb2d2c341e27166343ac" exitCode=0 Oct 30 22:06:08.866144 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:08.865809 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kb4qt" event={"ID":"03dcbb5f-dd7f-4ab6-ad37-4452802ea050","Type":"ContainerDied","Data":"e5b15a6702c2cd8418a4ebf3d013ad37043bd7db7a66eb2d2c341e27166343ac"} Oct 30 22:06:08.866144 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:08.865874 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kb4qt" event={"ID":"03dcbb5f-dd7f-4ab6-ad37-4452802ea050","Type":"ContainerStarted","Data":"d94c38a39571b9268d647364f0361fe0f083d93fb587015792a63e73ec2fadb8"} Oct 30 22:06:09.999532 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:09.999506 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kb4qt" Oct 30 22:06:10.155028 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:10.154955 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03dcbb5f-dd7f-4ab6-ad37-4452802ea050-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03dcbb5f-dd7f-4ab6-ad37-4452802ea050-maas-api-openshift-ca\") pod \"03dcbb5f-dd7f-4ab6-ad37-4452802ea050\" (UID: \"03dcbb5f-dd7f-4ab6-ad37-4452802ea050\") " Oct 30 22:06:10.155171 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:10.155047 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03dcbb5f-dd7f-4ab6-ad37-4452802ea050-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03dcbb5f-dd7f-4ab6-ad37-4452802ea050-maas-api-xks-ca\") pod \"03dcbb5f-dd7f-4ab6-ad37-4452802ea050\" (UID: \"03dcbb5f-dd7f-4ab6-ad37-4452802ea050\") " Oct 30 22:06:10.155171 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:10.155086 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03dcbb5f-dd7f-4ab6-ad37-4452802ea050-kube-api-access-g5ddf\" (UniqueName: \"kubernetes.io/projected/03dcbb5f-dd7f-4ab6-ad37-4452802ea050-kube-api-access-g5ddf\") pod \"03dcbb5f-dd7f-4ab6-ad37-4452802ea050\" (UID: \"03dcbb5f-dd7f-4ab6-ad37-4452802ea050\") " Oct 30 22:06:10.155317 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:10.155294 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03dcbb5f-dd7f-4ab6-ad37-4452802ea050-maas-api-openshift-ca" pod "03dcbb5f-dd7f-4ab6-ad37-4452802ea050" (UID: "03dcbb5f-dd7f-4ab6-ad37-4452802ea050"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:06:10.157134 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:10.157110 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03dcbb5f-dd7f-4ab6-ad37-4452802ea050-maas-api-xks-ca" pod "03dcbb5f-dd7f-4ab6-ad37-4452802ea050" (UID: "03dcbb5f-dd7f-4ab6-ad37-4452802ea050"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:06:10.157134 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:10.157121 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03dcbb5f-dd7f-4ab6-ad37-4452802ea050-kube-api-access-g5ddf" pod "03dcbb5f-dd7f-4ab6-ad37-4452802ea050" (UID: "03dcbb5f-dd7f-4ab6-ad37-4452802ea050"). InnerVolumeSpecName "kube-api-access-g5ddf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:10.256142 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:10.256112 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03dcbb5f-dd7f-4ab6-ad37-4452802ea050-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:10.256142 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:10.256139 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03dcbb5f-dd7f-4ab6-ad37-4452802ea050-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:10.256301 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:10.256152 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5ddf\" (UniqueName: \"kubernetes.io/projected/03dcbb5f-dd7f-4ab6-ad37-4452802ea050-kube-api-access-g5ddf\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:10.877187 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:10.877158 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kb4qt" Oct 30 22:06:10.877377 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:10.877188 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kb4qt" event={"ID":"03dcbb5f-dd7f-4ab6-ad37-4452802ea050","Type":"ContainerDied","Data":"d94c38a39571b9268d647364f0361fe0f083d93fb587015792a63e73ec2fadb8"} Oct 30 22:06:10.877377 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:10.877229 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d94c38a39571b9268d647364f0361fe0f083d93fb587015792a63e73ec2fadb8" Oct 30 22:06:19.001180 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.001139 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:19.460428 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.460396 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws"] Oct 30 22:06:19.460782 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.460748 5801 cpu_manager.go:414] "containerMap: removing container" podUID="03dcbb5f-dd7f-4ab6-ad37-4452802ea050" containerName="revoke-keys" Oct 30 22:06:19.460782 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.460778 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03dcbb5f-dd7f-4ab6-ad37-4452802ea050" containerName="revoke-keys" Oct 30 22:06:19.460889 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.460879 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="03dcbb5f-dd7f-4ab6-ad37-4452802ea050" containerName="revoke-keys" Oct 30 22:06:19.464134 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.464111 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.470188 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.470163 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-0a9006-openshift-default-dockercfg-n52gt\"" Oct 30 22:06:19.483176 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.483150 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws"] Oct 30 22:06:19.510450 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.510423 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:19.638137 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.638106 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-token\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.638311 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.638156 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzvzp\" (UniqueName: \"kubernetes.io/projected/e066ee75-6136-4867-b2b8-40f33b27c8bb-kube-api-access-tzvzp\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.638311 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.638259 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-envoy\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.638311 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.638296 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-credential-socket\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.638428 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.638394 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-podinfo\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.638479 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.638463 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-workload-certs\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.638526 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.638519 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-data\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.638598 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.638585 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-workload-socket\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.638687 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.638668 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e066ee75-6136-4867-b2b8-40f33b27c8bb-istiod-ca-cert\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.705289 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.705262 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:19.739583 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.739501 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-workload-socket\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.739583 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.739541 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e066ee75-6136-4867-b2b8-40f33b27c8bb-istiod-ca-cert\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.739583 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.739568 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-token\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.739877 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.739594 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzvzp\" (UniqueName: \"kubernetes.io/projected/e066ee75-6136-4867-b2b8-40f33b27c8bb-kube-api-access-tzvzp\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.739877 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.739715 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-envoy\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.739877 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.739786 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-credential-socket\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.739877 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.739846 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-podinfo\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.740109 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.739966 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-workload-socket\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.740109 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.740001 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-workload-certs\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.740109 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.740041 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-data\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.740243 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.740131 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-credential-socket\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.740355 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.740335 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e066ee75-6136-4867-b2b8-40f33b27c8bb-istiod-ca-cert\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.740415 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.740368 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-data\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.740415 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.740376 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-workload-certs\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.742158 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.742126 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-envoy\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.742338 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.742319 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-podinfo\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.752317 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.752288 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-token\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.752431 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.752327 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzvzp\" (UniqueName: \"kubernetes.io/projected/e066ee75-6136-4867-b2b8-40f33b27c8bb-kube-api-access-tzvzp\") pod \"e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:19.777396 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:19.777377 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:20.056550 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:20.056523 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws"] Oct 30 22:06:20.058567 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:06:20.058544 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode066ee75_6136_4867_b2b8_40f33b27c8bb.slice/crio-d1c07d42bdb4c0890276ad90c63798077eb8818e170479b3f5b963daedccf4dd WatchSource:0}: Error finding container d1c07d42bdb4c0890276ad90c63798077eb8818e170479b3f5b963daedccf4dd: Status 404 returned error can't find the container with id d1c07d42bdb4c0890276ad90c63798077eb8818e170479b3f5b963daedccf4dd Oct 30 22:06:20.060567 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:20.060538 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:06:20.060615 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:20.060596 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:06:20.060651 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:20.060627 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:06:20.926532 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:20.926490 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" event={"ID":"e066ee75-6136-4867-b2b8-40f33b27c8bb","Type":"ContainerStarted","Data":"8b63f21af2a9af6f56442cde1665e01082c2abc3dbddd1dd551d74beb9845cc5"} Oct 30 22:06:20.926701 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:20.926549 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" event={"ID":"e066ee75-6136-4867-b2b8-40f33b27c8bb","Type":"ContainerStarted","Data":"d1c07d42bdb4c0890276ad90c63798077eb8818e170479b3f5b963daedccf4dd"} Oct 30 22:06:20.952490 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:20.952445 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" podStartSLOduration=1.952433122 podStartE2EDuration="1.952433122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:20.9504456 +0000 UTC m=+2249.546262260" watchObservedRunningTime="2026-10-30 22:06:20.952433122 +0000 UTC m=+2249.548249782" Oct 30 22:06:21.778204 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:21.778168 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:21.783257 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:21.783231 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:21.932105 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:21.932070 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:21.932992 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:21.932967 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:23.293709 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:23.293673 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:23.401429 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:23.401394 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:24.506354 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.506318 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:24.576278 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.576243 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg"] Oct 30 22:06:24.580309 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.580289 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg" Oct 30 22:06:24.585567 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.585540 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz"] Oct 30 22:06:24.587926 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.587900 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc77743-0e00-47fa-a68c-51ca10c6a347-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg\" (UID: \"0fc77743-0e00-47fa-a68c-51ca10c6a347\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg" Oct 30 22:06:24.588136 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.588114 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z8xz\" (UniqueName: \"kubernetes.io/projected/0fc77743-0e00-47fa-a68c-51ca10c6a347-kube-api-access-9z8xz\") pod \"payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg\" (UID: \"0fc77743-0e00-47fa-a68c-51ca10c6a347\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg" Oct 30 22:06:24.589399 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.588986 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz" Oct 30 22:06:24.589953 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.589931 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-0a9006\"" Oct 30 22:06:24.590931 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.590909 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg"] Oct 30 22:06:24.598669 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.598645 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-559620a5-dockercfg-rcpjf\"" Oct 30 22:06:24.610275 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.610250 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz"] Oct 30 22:06:24.619269 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.619236 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:24.688707 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.688682 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f6086a-cbde-4d85-afab-15e1ad04f832-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz\" (UID: \"d3f6086a-cbde-4d85-afab-15e1ad04f832\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz" Oct 30 22:06:24.688844 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.688727 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z8xz\" (UniqueName: \"kubernetes.io/projected/0fc77743-0e00-47fa-a68c-51ca10c6a347-kube-api-access-9z8xz\") pod \"payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg\" (UID: \"0fc77743-0e00-47fa-a68c-51ca10c6a347\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg" Oct 30 22:06:24.688844 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.688766 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvmmr\" (UniqueName: \"kubernetes.io/projected/d3f6086a-cbde-4d85-afab-15e1ad04f832-kube-api-access-wvmmr\") pod \"payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz\" (UID: \"d3f6086a-cbde-4d85-afab-15e1ad04f832\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz" Oct 30 22:06:24.688930 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.688849 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc77743-0e00-47fa-a68c-51ca10c6a347-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg\" (UID: \"0fc77743-0e00-47fa-a68c-51ca10c6a347\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg" Oct 30 22:06:24.689541 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.689517 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc77743-0e00-47fa-a68c-51ca10c6a347-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg\" (UID: \"0fc77743-0e00-47fa-a68c-51ca10c6a347\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg" Oct 30 22:06:24.702628 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.702600 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z8xz\" (UniqueName: \"kubernetes.io/projected/0fc77743-0e00-47fa-a68c-51ca10c6a347-kube-api-access-9z8xz\") pod \"payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg\" (UID: \"0fc77743-0e00-47fa-a68c-51ca10c6a347\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg" Oct 30 22:06:24.717955 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.717920 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:24.790168 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.790075 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f6086a-cbde-4d85-afab-15e1ad04f832-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz\" (UID: \"d3f6086a-cbde-4d85-afab-15e1ad04f832\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz" Oct 30 22:06:24.790168 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.790129 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmmr\" (UniqueName: \"kubernetes.io/projected/d3f6086a-cbde-4d85-afab-15e1ad04f832-kube-api-access-wvmmr\") pod \"payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz\" (UID: \"d3f6086a-cbde-4d85-afab-15e1ad04f832\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz" Oct 30 22:06:24.790888 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.790862 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f6086a-cbde-4d85-afab-15e1ad04f832-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz\" (UID: \"d3f6086a-cbde-4d85-afab-15e1ad04f832\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz" Oct 30 22:06:24.802726 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.802701 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvmmr\" (UniqueName: \"kubernetes.io/projected/d3f6086a-cbde-4d85-afab-15e1ad04f832-kube-api-access-wvmmr\") pod \"payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz\" (UID: \"d3f6086a-cbde-4d85-afab-15e1ad04f832\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz" Oct 30 22:06:24.898471 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.898445 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg" Oct 30 22:06:24.913563 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:24.913540 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz" Oct 30 22:06:25.181455 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:25.181430 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg"] Oct 30 22:06:25.183665 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:06:25.183638 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc77743_0e00_47fa_a68c_51ca10c6a347.slice/crio-b33f52698cc3ff5260fafd7a09633f586a7ae3aa96edb51c939928256fc01559 WatchSource:0}: Error finding container b33f52698cc3ff5260fafd7a09633f586a7ae3aa96edb51c939928256fc01559: Status 404 returned error can't find the container with id b33f52698cc3ff5260fafd7a09633f586a7ae3aa96edb51c939928256fc01559 Oct 30 22:06:25.207570 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:25.207547 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz"] Oct 30 22:06:25.209166 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:06:25.209139 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f6086a_cbde_4d85_afab_15e1ad04f832.slice/crio-df4faeee937a430ad091902e3c2099e29e03d25bd4542b2d26af3598dbf4f943 WatchSource:0}: Error finding container df4faeee937a430ad091902e3c2099e29e03d25bd4542b2d26af3598dbf4f943: Status 404 returned error can't find the container with id df4faeee937a430ad091902e3c2099e29e03d25bd4542b2d26af3598dbf4f943 Oct 30 22:06:25.444995 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:25.444899 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:25.954474 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:25.954436 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz" event={"ID":"d3f6086a-cbde-4d85-afab-15e1ad04f832","Type":"ContainerStarted","Data":"7d2a8ae067118091ed27baea0c73f8540791ad27ecf0f4f70f7c2a920ec90be3"} Oct 30 22:06:25.954937 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:25.954493 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz" event={"ID":"d3f6086a-cbde-4d85-afab-15e1ad04f832","Type":"ContainerStarted","Data":"df4faeee937a430ad091902e3c2099e29e03d25bd4542b2d26af3598dbf4f943"} Oct 30 22:06:25.954937 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:25.954550 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz" Oct 30 22:06:25.956362 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:25.956339 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg" event={"ID":"0fc77743-0e00-47fa-a68c-51ca10c6a347","Type":"ContainerStarted","Data":"64ff6c68d44156800c540a216eeb666865817deb6f87f60a7a970e064ebf598a"} Oct 30 22:06:25.956481 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:25.956377 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg" event={"ID":"0fc77743-0e00-47fa-a68c-51ca10c6a347","Type":"ContainerStarted","Data":"b33f52698cc3ff5260fafd7a09633f586a7ae3aa96edb51c939928256fc01559"} Oct 30 22:06:25.956481 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:25.956421 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg" Oct 30 22:06:25.975798 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:25.975742 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz" podStartSLOduration=1.975730773 podStartE2EDuration="1.975730773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:25.973846053 +0000 UTC m=+2254.569662974" watchObservedRunningTime="2026-10-30 22:06:25.975730773 +0000 UTC m=+2254.571547435" Oct 30 22:06:25.996023 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:25.995982 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg" podStartSLOduration=1.99597125 podStartE2EDuration="1.99597125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:25.994844686 +0000 UTC m=+2254.590661370" watchObservedRunningTime="2026-10-30 22:06:25.99597125 +0000 UTC m=+2254.591787910" Oct 30 22:06:26.185656 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:26.185625 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:27.099540 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:27.099498 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:27.847077 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:27.847041 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:28.579576 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:28.579539 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:29.478370 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:29.478334 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:30.194656 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:30.194623 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:30.297327 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:06:30.297291 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:06:30.903609 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:30.903570 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:31.585728 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:31.585696 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:32.289689 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:32.289649 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:33.027711 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:33.027675 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:33.777769 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:33.777711 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:34.491889 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:34.491846 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:35.187992 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:35.187960 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:35.970194 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:35.970151 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:36.025079 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:36.024998 5801 generic.go:356] "Generic (PLEG): container finished" podID="e066ee75-6136-4867-b2b8-40f33b27c8bb" containerID="8b63f21af2a9af6f56442cde1665e01082c2abc3dbddd1dd551d74beb9845cc5" exitCode=0 Oct 30 22:06:36.025079 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:36.025069 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" event={"ID":"e066ee75-6136-4867-b2b8-40f33b27c8bb","Type":"ContainerDied","Data":"8b63f21af2a9af6f56442cde1665e01082c2abc3dbddd1dd551d74beb9845cc5"} Oct 30 22:06:36.025484 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:36.025468 5801 scope.go:122] "RemoveContainer" containerID="8b63f21af2a9af6f56442cde1665e01082c2abc3dbddd1dd551d74beb9845cc5" Oct 30 22:06:36.026707 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:36.026681 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:06:36.026805 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:36.026790 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:06:36.026875 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:36.026834 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:06:36.785925 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:36.785890 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:36.932295 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:36.932263 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:36.978247 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:36.978216 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz" Oct 30 22:06:36.978669 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:36.978398 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg" Oct 30 22:06:37.032331 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:37.032292 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" event={"ID":"e066ee75-6136-4867-b2b8-40f33b27c8bb","Type":"ContainerStarted","Data":"5bdfeb49e8604cae0ae657a553e496db73b54690ccc5966fa2d9b06e55848957"} Oct 30 22:06:37.676791 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:37.676718 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:37.778384 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:37.778343 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:37.779713 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:37.779685 5801 patch_prober.go:29] interesting pod/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body= Oct 30 22:06:37.779855 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:37.779737 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" podUID="e066ee75-6136-4867-b2b8-40f33b27c8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" Oct 30 22:06:38.385999 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:38.385965 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:38.489811 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:38.489774 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:38.778240 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:38.778149 5801 patch_prober.go:29] interesting pod/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body= Oct 30 22:06:38.778398 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:38.778234 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" podUID="e066ee75-6136-4867-b2b8-40f33b27c8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" Oct 30 22:06:39.074997 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:39.074964 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:39.205663 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:39.205632 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:39.345783 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:39.345684 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:39.778832 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:39.778710 5801 patch_prober.go:29] interesting pod/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body= Oct 30 22:06:39.778832 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:39.778821 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" podUID="e066ee75-6136-4867-b2b8-40f33b27c8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" Oct 30 22:06:40.014016 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:40.013975 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:40.244512 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:40.244475 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:40.778201 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:40.778153 5801 patch_prober.go:29] interesting pod/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body= Oct 30 22:06:40.778411 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:40.778250 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" podUID="e066ee75-6136-4867-b2b8-40f33b27c8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" Oct 30 22:06:41.119706 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:41.119673 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:41.778207 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:41.778155 5801 patch_prober.go:29] interesting pod/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body= Oct 30 22:06:41.778431 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:41.778235 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" podUID="e066ee75-6136-4867-b2b8-40f33b27c8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" Oct 30 22:06:41.878048 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:41.878000 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:42.782996 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:42.782959 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:42.783407 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:42.783268 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:42.784101 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:42.784082 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:06:42.872253 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:42.872209 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:43.576200 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:43.576165 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:44.377810 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:44.377773 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:45.228021 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:45.227984 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:45.974329 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:45.974291 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:46.783207 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:46.783167 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:47.490153 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:47.490116 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:48.183658 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:48.183625 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:48.875695 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:48.875657 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:49.677734 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:49.677691 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:50.471905 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:50.471869 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:51.177299 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:51.177262 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:51.875711 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:51.875661 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:52.680914 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:52.680875 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:53.394092 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:53.394056 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:54.077803 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:54.077748 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:54.788229 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:54.788196 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:55.637466 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:55.637368 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:56.406500 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:56.406464 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:57.272562 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:57.272503 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:57.979952 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:57.979921 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:58.931150 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:58.931115 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:59.189160 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:59.189061 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:59.793639 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:06:59.793607 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:00.277394 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:07:00.277359 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:07:00.528129 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:00.528026 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:01.313527 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:01.313491 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:01.837332 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:01.837295 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:02.038037 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:02.037996 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:02.138371 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:02.138277 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:02.781118 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:02.781088 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:03.619397 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:03.619360 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:04.281979 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:04.281947 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:04.978548 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:04.978502 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:05.682641 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:05.682609 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:06.382510 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:06.382466 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:07.088069 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:07.088026 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:07.783850 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:07.783813 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:08.491619 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:08.491584 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:09.249838 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:09.249804 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:09.880028 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:09.879994 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:10.573201 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:10.573164 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:11.184738 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:11.184702 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d98392e-hm2tb"] Oct 30 22:07:11.189459 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:11.189440 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d98392e-hm2tb" Oct 30 22:07:11.203446 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:11.203418 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:07:11.203659 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:11.203600 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 22:07:11.208134 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:11.208109 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d98392e-hm2tb"] Oct 30 22:07:11.229656 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:11.229635 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:07:11.229851 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:11.229833 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:07:11.289828 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:11.289791 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/467e3901-d46d-4721-a24c-ee97d242e806-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d98392e-hm2tb\" (UID: \"467e3901-d46d-4721-a24c-ee97d242e806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d98392e-hm2tb" Oct 30 22:07:11.290021 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:11.289905 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nbgt\" (UniqueName: \"kubernetes.io/projected/467e3901-d46d-4721-a24c-ee97d242e806-kube-api-access-4nbgt\") pod \"maas-api-revoke-sub-5d98392e-hm2tb\" (UID: \"467e3901-d46d-4721-a24c-ee97d242e806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d98392e-hm2tb" Oct 30 22:07:11.290021 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:11.289957 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/467e3901-d46d-4721-a24c-ee97d242e806-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d98392e-hm2tb\" (UID: \"467e3901-d46d-4721-a24c-ee97d242e806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d98392e-hm2tb" Oct 30 22:07:11.391364 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:11.391328 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nbgt\" (UniqueName: \"kubernetes.io/projected/467e3901-d46d-4721-a24c-ee97d242e806-kube-api-access-4nbgt\") pod \"maas-api-revoke-sub-5d98392e-hm2tb\" (UID: \"467e3901-d46d-4721-a24c-ee97d242e806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d98392e-hm2tb" Oct 30 22:07:11.391638 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:11.391400 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/467e3901-d46d-4721-a24c-ee97d242e806-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d98392e-hm2tb\" (UID: \"467e3901-d46d-4721-a24c-ee97d242e806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d98392e-hm2tb" Oct 30 22:07:11.391638 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:11.391511 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/467e3901-d46d-4721-a24c-ee97d242e806-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d98392e-hm2tb\" (UID: \"467e3901-d46d-4721-a24c-ee97d242e806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d98392e-hm2tb" Oct 30 22:07:11.392220 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:11.392189 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/467e3901-d46d-4721-a24c-ee97d242e806-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d98392e-hm2tb\" (UID: \"467e3901-d46d-4721-a24c-ee97d242e806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d98392e-hm2tb" Oct 30 22:07:11.393885 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:11.393855 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/467e3901-d46d-4721-a24c-ee97d242e806-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d98392e-hm2tb\" (UID: \"467e3901-d46d-4721-a24c-ee97d242e806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d98392e-hm2tb" Oct 30 22:07:11.403267 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:11.403246 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nbgt\" (UniqueName: \"kubernetes.io/projected/467e3901-d46d-4721-a24c-ee97d242e806-kube-api-access-4nbgt\") pod \"maas-api-revoke-sub-5d98392e-hm2tb\" (UID: \"467e3901-d46d-4721-a24c-ee97d242e806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d98392e-hm2tb" Oct 30 22:07:11.477322 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:11.477243 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:11.513226 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:11.513200 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d98392e-hm2tb" Oct 30 22:07:11.794619 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:11.794591 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d98392e-hm2tb"] Oct 30 22:07:11.796383 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:07:11.796355 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467e3901_d46d_4721_a24c_ee97d242e806.slice/crio-3e5d365125010ad7539b0bea15700a95f6dd06276eec19ba297032fb802eb971 WatchSource:0}: Error finding container 3e5d365125010ad7539b0bea15700a95f6dd06276eec19ba297032fb802eb971: Status 404 returned error can't find the container with id 3e5d365125010ad7539b0bea15700a95f6dd06276eec19ba297032fb802eb971 Oct 30 22:07:12.182849 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:12.182817 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:12.245942 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:12.245906 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d98392e-hm2tb" event={"ID":"467e3901-d46d-4721-a24c-ee97d242e806","Type":"ContainerStarted","Data":"a7c6ea7e816d9706c59a1b480bc15eb0a161da5cddb8b43ba9a8f7ecc6e83b20"} Oct 30 22:07:12.246386 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:12.245961 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d98392e-hm2tb" event={"ID":"467e3901-d46d-4721-a24c-ee97d242e806","Type":"ContainerStarted","Data":"3e5d365125010ad7539b0bea15700a95f6dd06276eec19ba297032fb802eb971"} Oct 30 22:07:12.268528 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:12.268483 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d98392e-hm2tb" podStartSLOduration=1.268471057 podStartE2EDuration="1.268471057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:12.265986198 +0000 UTC m=+2300.861802858" watchObservedRunningTime="2026-10-30 22:07:12.268471057 +0000 UTC m=+2300.864287754" Oct 30 22:07:12.978433 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:12.978349 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:13.252375 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:13.252288 5801 generic.go:356] "Generic (PLEG): container finished" podID="467e3901-d46d-4721-a24c-ee97d242e806" containerID="a7c6ea7e816d9706c59a1b480bc15eb0a161da5cddb8b43ba9a8f7ecc6e83b20" exitCode=0 Oct 30 22:07:13.252795 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:13.252378 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d98392e-hm2tb" event={"ID":"467e3901-d46d-4721-a24c-ee97d242e806","Type":"ContainerDied","Data":"a7c6ea7e816d9706c59a1b480bc15eb0a161da5cddb8b43ba9a8f7ecc6e83b20"} Oct 30 22:07:13.679966 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:13.679926 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:14.381161 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:14.381132 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:14.390321 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:14.390298 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d98392e-hm2tb" Oct 30 22:07:14.521431 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:14.521349 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/467e3901-d46d-4721-a24c-ee97d242e806-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/467e3901-d46d-4721-a24c-ee97d242e806-maas-api-openshift-ca\") pod \"467e3901-d46d-4721-a24c-ee97d242e806\" (UID: \"467e3901-d46d-4721-a24c-ee97d242e806\") " Oct 30 22:07:14.521431 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:14.521386 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467e3901-d46d-4721-a24c-ee97d242e806-kube-api-access-4nbgt\" (UniqueName: \"kubernetes.io/projected/467e3901-d46d-4721-a24c-ee97d242e806-kube-api-access-4nbgt\") pod \"467e3901-d46d-4721-a24c-ee97d242e806\" (UID: \"467e3901-d46d-4721-a24c-ee97d242e806\") " Oct 30 22:07:14.521653 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:14.521514 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/467e3901-d46d-4721-a24c-ee97d242e806-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/467e3901-d46d-4721-a24c-ee97d242e806-maas-api-xks-ca\") pod \"467e3901-d46d-4721-a24c-ee97d242e806\" (UID: \"467e3901-d46d-4721-a24c-ee97d242e806\") " Oct 30 22:07:14.521727 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:14.521704 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/467e3901-d46d-4721-a24c-ee97d242e806-maas-api-openshift-ca" pod "467e3901-d46d-4721-a24c-ee97d242e806" (UID: "467e3901-d46d-4721-a24c-ee97d242e806"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:14.521942 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:14.521921 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/467e3901-d46d-4721-a24c-ee97d242e806-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:14.523571 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:14.523550 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467e3901-d46d-4721-a24c-ee97d242e806-kube-api-access-4nbgt" pod "467e3901-d46d-4721-a24c-ee97d242e806" (UID: "467e3901-d46d-4721-a24c-ee97d242e806"). InnerVolumeSpecName "kube-api-access-4nbgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:14.523571 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:14.523550 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/467e3901-d46d-4721-a24c-ee97d242e806-maas-api-xks-ca" pod "467e3901-d46d-4721-a24c-ee97d242e806" (UID: "467e3901-d46d-4721-a24c-ee97d242e806"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:14.622456 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:14.622425 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nbgt\" (UniqueName: \"kubernetes.io/projected/467e3901-d46d-4721-a24c-ee97d242e806-kube-api-access-4nbgt\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:14.622456 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:14.622453 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/467e3901-d46d-4721-a24c-ee97d242e806-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:15.077667 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:15.077630 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:15.263948 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:15.263917 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d98392e-hm2tb" Oct 30 22:07:15.264150 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:15.263949 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d98392e-hm2tb" event={"ID":"467e3901-d46d-4721-a24c-ee97d242e806","Type":"ContainerDied","Data":"3e5d365125010ad7539b0bea15700a95f6dd06276eec19ba297032fb802eb971"} Oct 30 22:07:15.264150 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:15.264007 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e5d365125010ad7539b0bea15700a95f6dd06276eec19ba297032fb802eb971" Oct 30 22:07:15.893163 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:15.893127 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:16.585318 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:16.585270 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:17.402451 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:17.402414 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:18.085169 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:18.085133 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:18.780552 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:18.780511 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:19.474529 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:19.474491 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:20.190088 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:20.190045 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:20.871426 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:20.871390 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:21.582935 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:21.582904 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:22.030023 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:22.029929 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:22.243998 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:22.243961 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs"] Oct 30 22:07:22.244389 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:22.244375 5801 cpu_manager.go:414] "containerMap: removing container" podUID="467e3901-d46d-4721-a24c-ee97d242e806" containerName="revoke-keys" Oct 30 22:07:22.244437 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:22.244391 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467e3901-d46d-4721-a24c-ee97d242e806" containerName="revoke-keys" Oct 30 22:07:22.244490 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:22.244481 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="467e3901-d46d-4721-a24c-ee97d242e806" containerName="revoke-keys" Oct 30 22:07:22.247545 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:22.247527 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs" Oct 30 22:07:22.282935 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:22.282882 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:07:22.287387 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:22.287366 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs"] Oct 30 22:07:22.290835 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:22.290815 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:07:22.290966 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:22.290953 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 22:07:22.339883 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:22.339848 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:07:22.403416 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:22.403389 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4snr2\" (UniqueName: \"kubernetes.io/projected/a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d-kube-api-access-4snr2\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs\" (UID: \"a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs" Oct 30 22:07:22.403573 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:22.403452 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs\" (UID: \"a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs" Oct 30 22:07:22.403642 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:22.403619 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs\" (UID: \"a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs" Oct 30 22:07:22.422406 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:22.422378 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:22.504273 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:22.504241 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4snr2\" (UniqueName: \"kubernetes.io/projected/a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d-kube-api-access-4snr2\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs\" (UID: \"a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs" Oct 30 22:07:22.504475 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:22.504322 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs\" (UID: \"a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs" Oct 30 22:07:22.504475 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:22.504433 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs\" (UID: \"a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs" Oct 30 22:07:22.505102 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:22.505076 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs\" (UID: \"a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs" Oct 30 22:07:22.506711 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:22.506689 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs\" (UID: \"a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs" Oct 30 22:07:22.516611 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:22.516593 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4snr2\" (UniqueName: \"kubernetes.io/projected/a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d-kube-api-access-4snr2\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs\" (UID: \"a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs" Oct 30 22:07:22.560386 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:22.560356 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs" Oct 30 22:07:22.835901 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:22.835871 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs"] Oct 30 22:07:22.837569 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:07:22.837540 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e5ce5a_dadb_472c_9fff_f2fa0a06cc2d.slice/crio-a6b5f0a169bfff00f4ef7c576a6c6dfdda8b273cf439a753549f9a09cbc91e9c WatchSource:0}: Error finding container a6b5f0a169bfff00f4ef7c576a6c6dfdda8b273cf439a753549f9a09cbc91e9c: Status 404 returned error can't find the container with id a6b5f0a169bfff00f4ef7c576a6c6dfdda8b273cf439a753549f9a09cbc91e9c Oct 30 22:07:23.173365 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:23.173324 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:23.308359 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:23.308321 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs" event={"ID":"a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d","Type":"ContainerStarted","Data":"c6cf0bc7b4c186a381613b4ccb76d3ab34a2a405ba26c0d3259eda341a06c519"} Oct 30 22:07:23.308538 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:23.308378 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs" event={"ID":"a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d","Type":"ContainerStarted","Data":"a6b5f0a169bfff00f4ef7c576a6c6dfdda8b273cf439a753549f9a09cbc91e9c"} Oct 30 22:07:23.328403 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:23.328344 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs" podStartSLOduration=1.328330965 podStartE2EDuration="1.328330965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:23.326157416 +0000 UTC m=+2311.921974079" watchObservedRunningTime="2026-10-30 22:07:23.328330965 +0000 UTC m=+2311.924147626" Oct 30 22:07:23.978128 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:23.978040 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:24.316214 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:24.316174 5801 generic.go:356] "Generic (PLEG): container finished" podID="a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d" containerID="c6cf0bc7b4c186a381613b4ccb76d3ab34a2a405ba26c0d3259eda341a06c519" exitCode=0 Oct 30 22:07:24.316376 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:24.316258 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs" event={"ID":"a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d","Type":"ContainerDied","Data":"c6cf0bc7b4c186a381613b4ccb76d3ab34a2a405ba26c0d3259eda341a06c519"} Oct 30 22:07:24.755550 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:24.755461 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:25.461768 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:25.461726 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs" Oct 30 22:07:25.484667 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:25.484632 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:25.635828 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:25.635700 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d-maas-api-openshift-ca\") pod \"a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d\" (UID: \"a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d\") " Oct 30 22:07:25.635828 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:25.635798 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d-maas-api-xks-ca\") pod \"a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d\" (UID: \"a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d\") " Oct 30 22:07:25.636091 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:25.635881 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d-kube-api-access-4snr2\" (UniqueName: \"kubernetes.io/projected/a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d-kube-api-access-4snr2\") pod \"a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d\" (UID: \"a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d\") " Oct 30 22:07:25.636195 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:25.636168 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d-maas-api-openshift-ca" pod "a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d" (UID: "a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:25.638044 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:25.638014 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d-maas-api-xks-ca" pod "a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d" (UID: "a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:25.638155 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:25.638078 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d-kube-api-access-4snr2" pod "a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d" (UID: "a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d"). InnerVolumeSpecName "kube-api-access-4snr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:25.737042 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:25.737010 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:25.737042 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:25.737039 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:25.737260 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:25.737051 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4snr2\" (UniqueName: \"kubernetes.io/projected/a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d-kube-api-access-4snr2\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:26.181291 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:26.181253 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:26.329115 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:26.329089 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs" Oct 30 22:07:26.329293 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:26.329085 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs" event={"ID":"a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d","Type":"ContainerDied","Data":"a6b5f0a169bfff00f4ef7c576a6c6dfdda8b273cf439a753549f9a09cbc91e9c"} Oct 30 22:07:26.329293 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:26.329215 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6b5f0a169bfff00f4ef7c576a6c6dfdda8b273cf439a753549f9a09cbc91e9c" Oct 30 22:07:26.878057 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:26.878026 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:27.079634 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:27.079592 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:27.570547 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:27.570500 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:28.321296 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:28.321254 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:28.986657 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:28.986620 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:29.759180 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:29.759144 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:30.277904 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:07:30.277866 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:07:30.387866 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:30.387824 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:31.086524 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:31.086468 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:31.768359 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:31.768324 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:32.383525 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:32.383485 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz"] Oct 30 22:07:32.387505 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:32.387470 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg"] Oct 30 22:07:32.387665 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:32.387619 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:32.488474 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:32.488438 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:32.752490 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:32.752401 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:32.778389 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:32.778361 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs"] Oct 30 22:07:32.781989 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:32.781960 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a9006-q4lbs"] Oct 30 22:07:32.847020 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:32.846984 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:33.461379 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:33.461345 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:33.528535 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:33.524446 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d" path="/var/lib/kubelet/pods/a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d/volumes" Oct 30 22:07:37.384870 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:37.384828 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz" podUID="d3f6086a-cbde-4d85-afab-15e1ad04f832" containerName="payload-processing" containerID="cri-o://7d2a8ae067118091ed27baea0c73f8540791ad27ecf0f4f70f7c2a920ec90be3" gracePeriod=25 Oct 30 22:07:37.388656 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:37.388627 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg" podUID="0fc77743-0e00-47fa-a68c-51ca10c6a347" containerName="payload-pre-processing" containerID="cri-o://64ff6c68d44156800c540a216eeb666865817deb6f87f60a7a970e064ebf598a" gracePeriod=25 Oct 30 22:07:39.631912 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:39.631877 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:39.730385 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:39.730337 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:39.976810 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:39.976705 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws"] Oct 30 22:07:39.977300 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:39.977268 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" podUID="e066ee75-6136-4867-b2b8-40f33b27c8bb" containerName="istio-proxy" containerID="cri-o://5bdfeb49e8604cae0ae657a553e496db73b54690ccc5966fa2d9b06e55848957" gracePeriod=30 Oct 30 22:07:40.050846 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:40.050797 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:42.785088 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:42.785054 5801 patch_prober.go:29] interesting pod/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:07:42.785473 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:42.785119 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" podUID="e066ee75-6136-4867-b2b8-40f33b27c8bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:07:44.231406 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:44.231367 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:45.229959 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.229933 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:07:45.314053 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.314027 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e066ee75-6136-4867-b2b8-40f33b27c8bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e066ee75-6136-4867-b2b8-40f33b27c8bb-istiod-ca-cert\") pod \"e066ee75-6136-4867-b2b8-40f33b27c8bb\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " Oct 30 22:07:45.314465 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.314059 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-token\" (UniqueName: \"kubernetes.io/projected/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-token\") pod \"e066ee75-6136-4867-b2b8-40f33b27c8bb\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " Oct 30 22:07:45.314465 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.314106 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-envoy\") pod \"e066ee75-6136-4867-b2b8-40f33b27c8bb\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " Oct 30 22:07:45.314465 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.314129 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-workload-socket\") pod \"e066ee75-6136-4867-b2b8-40f33b27c8bb\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " Oct 30 22:07:45.314465 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.314152 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-workload-certs\") pod \"e066ee75-6136-4867-b2b8-40f33b27c8bb\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " Oct 30 22:07:45.314465 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.314190 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-podinfo\") pod \"e066ee75-6136-4867-b2b8-40f33b27c8bb\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " Oct 30 22:07:45.314465 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.314213 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-data\") pod \"e066ee75-6136-4867-b2b8-40f33b27c8bb\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " Oct 30 22:07:45.314465 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.314264 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-credential-socket\") pod \"e066ee75-6136-4867-b2b8-40f33b27c8bb\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " Oct 30 22:07:45.314465 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.314303 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e066ee75-6136-4867-b2b8-40f33b27c8bb-kube-api-access-tzvzp\" (UniqueName: \"kubernetes.io/projected/e066ee75-6136-4867-b2b8-40f33b27c8bb-kube-api-access-tzvzp\") pod \"e066ee75-6136-4867-b2b8-40f33b27c8bb\" (UID: \"e066ee75-6136-4867-b2b8-40f33b27c8bb\") " Oct 30 22:07:45.314929 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.314529 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-workload-certs" pod "e066ee75-6136-4867-b2b8-40f33b27c8bb" (UID: "e066ee75-6136-4867-b2b8-40f33b27c8bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:45.314929 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.314545 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-workload-socket" pod "e066ee75-6136-4867-b2b8-40f33b27c8bb" (UID: "e066ee75-6136-4867-b2b8-40f33b27c8bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:45.314929 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.314551 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e066ee75-6136-4867-b2b8-40f33b27c8bb-istiod-ca-cert" pod "e066ee75-6136-4867-b2b8-40f33b27c8bb" (UID: "e066ee75-6136-4867-b2b8-40f33b27c8bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:45.314929 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.314671 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-data" pod "e066ee75-6136-4867-b2b8-40f33b27c8bb" (UID: "e066ee75-6136-4867-b2b8-40f33b27c8bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:45.314929 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.314800 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e066ee75-6136-4867-b2b8-40f33b27c8bb-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:45.314929 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.314822 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:45.314929 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.314838 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:45.314929 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.314837 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-credential-socket" pod "e066ee75-6136-4867-b2b8-40f33b27c8bb" (UID: "e066ee75-6136-4867-b2b8-40f33b27c8bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:45.316566 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.316544 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-podinfo" pod "e066ee75-6136-4867-b2b8-40f33b27c8bb" (UID: "e066ee75-6136-4867-b2b8-40f33b27c8bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:07:45.316693 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.316670 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e066ee75-6136-4867-b2b8-40f33b27c8bb-kube-api-access-tzvzp" pod "e066ee75-6136-4867-b2b8-40f33b27c8bb" (UID: "e066ee75-6136-4867-b2b8-40f33b27c8bb"). InnerVolumeSpecName "kube-api-access-tzvzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:45.316849 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.316746 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-envoy" pod "e066ee75-6136-4867-b2b8-40f33b27c8bb" (UID: "e066ee75-6136-4867-b2b8-40f33b27c8bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:45.316849 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.316784 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-token" pod "e066ee75-6136-4867-b2b8-40f33b27c8bb" (UID: "e066ee75-6136-4867-b2b8-40f33b27c8bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:45.416061 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.416028 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:45.416061 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.416051 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:45.416061 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.416062 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:45.416285 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.416071 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzvzp\" (UniqueName: \"kubernetes.io/projected/e066ee75-6136-4867-b2b8-40f33b27c8bb-kube-api-access-tzvzp\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:45.416285 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.416082 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:45.416285 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.416091 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e066ee75-6136-4867-b2b8-40f33b27c8bb-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:45.429917 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.429890 5801 generic.go:356] "Generic (PLEG): container finished" podID="e066ee75-6136-4867-b2b8-40f33b27c8bb" containerID="5bdfeb49e8604cae0ae657a553e496db73b54690ccc5966fa2d9b06e55848957" exitCode=0 Oct 30 22:07:45.430039 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.429938 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" Oct 30 22:07:45.430039 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.429985 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" event={"ID":"e066ee75-6136-4867-b2b8-40f33b27c8bb","Type":"ContainerDied","Data":"5bdfeb49e8604cae0ae657a553e496db73b54690ccc5966fa2d9b06e55848957"} Oct 30 22:07:45.430118 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.430044 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws" event={"ID":"e066ee75-6136-4867-b2b8-40f33b27c8bb","Type":"ContainerDied","Data":"d1c07d42bdb4c0890276ad90c63798077eb8818e170479b3f5b963daedccf4dd"} Oct 30 22:07:45.430118 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.430068 5801 scope.go:122] "RemoveContainer" containerID="5bdfeb49e8604cae0ae657a553e496db73b54690ccc5966fa2d9b06e55848957" Oct 30 22:07:45.442554 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.442532 5801 scope.go:122] "RemoveContainer" containerID="8b63f21af2a9af6f56442cde1665e01082c2abc3dbddd1dd551d74beb9845cc5" Oct 30 22:07:45.450970 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.450955 5801 scope.go:122] "RemoveContainer" containerID="5bdfeb49e8604cae0ae657a553e496db73b54690ccc5966fa2d9b06e55848957" Oct 30 22:07:45.451196 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:07:45.451174 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bdfeb49e8604cae0ae657a553e496db73b54690ccc5966fa2d9b06e55848957\": container with ID starting with 5bdfeb49e8604cae0ae657a553e496db73b54690ccc5966fa2d9b06e55848957 not found: ID does not exist" containerID="5bdfeb49e8604cae0ae657a553e496db73b54690ccc5966fa2d9b06e55848957" Oct 30 22:07:45.451269 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.451208 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bdfeb49e8604cae0ae657a553e496db73b54690ccc5966fa2d9b06e55848957"} err="failed to get container status \"5bdfeb49e8604cae0ae657a553e496db73b54690ccc5966fa2d9b06e55848957\": rpc error: code = NotFound desc = could not find container \"5bdfeb49e8604cae0ae657a553e496db73b54690ccc5966fa2d9b06e55848957\": container with ID starting with 5bdfeb49e8604cae0ae657a553e496db73b54690ccc5966fa2d9b06e55848957 not found: ID does not exist" Oct 30 22:07:45.451269 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.451233 5801 scope.go:122] "RemoveContainer" containerID="8b63f21af2a9af6f56442cde1665e01082c2abc3dbddd1dd551d74beb9845cc5" Oct 30 22:07:45.451493 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:07:45.451473 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b63f21af2a9af6f56442cde1665e01082c2abc3dbddd1dd551d74beb9845cc5\": container with ID starting with 8b63f21af2a9af6f56442cde1665e01082c2abc3dbddd1dd551d74beb9845cc5 not found: ID does not exist" containerID="8b63f21af2a9af6f56442cde1665e01082c2abc3dbddd1dd551d74beb9845cc5" Oct 30 22:07:45.451539 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.451500 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b63f21af2a9af6f56442cde1665e01082c2abc3dbddd1dd551d74beb9845cc5"} err="failed to get container status \"8b63f21af2a9af6f56442cde1665e01082c2abc3dbddd1dd551d74beb9845cc5\": rpc error: code = NotFound desc = could not find container \"8b63f21af2a9af6f56442cde1665e01082c2abc3dbddd1dd551d74beb9845cc5\": container with ID starting with 8b63f21af2a9af6f56442cde1665e01082c2abc3dbddd1dd551d74beb9845cc5 not found: ID does not exist" Oct 30 22:07:45.458857 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.458834 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws"] Oct 30 22:07:45.461556 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.461537 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0a9006-openshift-default-69bdd97fb9v9cws"] Oct 30 22:07:45.522555 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:45.522534 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e066ee75-6136-4867-b2b8-40f33b27c8bb" path="/var/lib/kubelet/pods/e066ee75-6136-4867-b2b8-40f33b27c8bb/volumes" Oct 30 22:07:46.963976 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:46.963931 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz" podUID="d3f6086a-cbde-4d85-afab-15e1ad04f832" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:07:46.964499 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:46.964040 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg" podUID="0fc77743-0e00-47fa-a68c-51ca10c6a347" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:07:56.963348 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:56.963310 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz" podUID="d3f6086a-cbde-4d85-afab-15e1ad04f832" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:07:56.963348 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:56.963337 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg" podUID="0fc77743-0e00-47fa-a68c-51ca10c6a347" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:07:57.533316 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:57.533278 5801 generic.go:356] "Generic (PLEG): container finished" podID="0fc77743-0e00-47fa-a68c-51ca10c6a347" containerID="64ff6c68d44156800c540a216eeb666865817deb6f87f60a7a970e064ebf598a" exitCode=0 Oct 30 22:07:57.533889 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:57.533560 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg" event={"ID":"0fc77743-0e00-47fa-a68c-51ca10c6a347","Type":"ContainerDied","Data":"64ff6c68d44156800c540a216eeb666865817deb6f87f60a7a970e064ebf598a"} Oct 30 22:07:57.535832 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:57.535703 5801 generic.go:356] "Generic (PLEG): container finished" podID="d3f6086a-cbde-4d85-afab-15e1ad04f832" containerID="7d2a8ae067118091ed27baea0c73f8540791ad27ecf0f4f70f7c2a920ec90be3" exitCode=0 Oct 30 22:07:57.535832 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:57.535731 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz" event={"ID":"d3f6086a-cbde-4d85-afab-15e1ad04f832","Type":"ContainerDied","Data":"7d2a8ae067118091ed27baea0c73f8540791ad27ecf0f4f70f7c2a920ec90be3"} Oct 30 22:07:57.642662 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:57.642604 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz" Oct 30 22:07:57.668715 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:57.668691 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg" Oct 30 22:07:57.684240 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:57.684207 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f6086a-cbde-4d85-afab-15e1ad04f832-kube-api-access-wvmmr\" (UniqueName: \"kubernetes.io/projected/d3f6086a-cbde-4d85-afab-15e1ad04f832-kube-api-access-wvmmr\") pod \"d3f6086a-cbde-4d85-afab-15e1ad04f832\" (UID: \"d3f6086a-cbde-4d85-afab-15e1ad04f832\") " Oct 30 22:07:57.684374 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:57.684272 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3f6086a-cbde-4d85-afab-15e1ad04f832-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f6086a-cbde-4d85-afab-15e1ad04f832-plugins-config-volume\") pod \"d3f6086a-cbde-4d85-afab-15e1ad04f832\" (UID: \"d3f6086a-cbde-4d85-afab-15e1ad04f832\") " Oct 30 22:07:57.684604 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:57.684579 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f6086a-cbde-4d85-afab-15e1ad04f832-plugins-config-volume" pod "d3f6086a-cbde-4d85-afab-15e1ad04f832" (UID: "d3f6086a-cbde-4d85-afab-15e1ad04f832"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:57.686310 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:57.686288 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f6086a-cbde-4d85-afab-15e1ad04f832-kube-api-access-wvmmr" pod "d3f6086a-cbde-4d85-afab-15e1ad04f832" (UID: "d3f6086a-cbde-4d85-afab-15e1ad04f832"). InnerVolumeSpecName "kube-api-access-wvmmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:57.785564 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:57.785536 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fc77743-0e00-47fa-a68c-51ca10c6a347-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc77743-0e00-47fa-a68c-51ca10c6a347-plugins-config-volume\") pod \"0fc77743-0e00-47fa-a68c-51ca10c6a347\" (UID: \"0fc77743-0e00-47fa-a68c-51ca10c6a347\") " Oct 30 22:07:57.785699 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:57.785623 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc77743-0e00-47fa-a68c-51ca10c6a347-kube-api-access-9z8xz\" (UniqueName: \"kubernetes.io/projected/0fc77743-0e00-47fa-a68c-51ca10c6a347-kube-api-access-9z8xz\") pod \"0fc77743-0e00-47fa-a68c-51ca10c6a347\" (UID: \"0fc77743-0e00-47fa-a68c-51ca10c6a347\") " Oct 30 22:07:57.785873 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:57.785852 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fc77743-0e00-47fa-a68c-51ca10c6a347-plugins-config-volume" pod "0fc77743-0e00-47fa-a68c-51ca10c6a347" (UID: "0fc77743-0e00-47fa-a68c-51ca10c6a347"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:57.785957 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:57.785944 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvmmr\" (UniqueName: \"kubernetes.io/projected/d3f6086a-cbde-4d85-afab-15e1ad04f832-kube-api-access-wvmmr\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:57.786003 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:57.785960 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f6086a-cbde-4d85-afab-15e1ad04f832-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:57.786003 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:57.785970 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc77743-0e00-47fa-a68c-51ca10c6a347-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:57.787386 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:57.787366 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc77743-0e00-47fa-a68c-51ca10c6a347-kube-api-access-9z8xz" pod "0fc77743-0e00-47fa-a68c-51ca10c6a347" (UID: "0fc77743-0e00-47fa-a68c-51ca10c6a347"). InnerVolumeSpecName "kube-api-access-9z8xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:57.886427 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:57.886382 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z8xz\" (UniqueName: \"kubernetes.io/projected/0fc77743-0e00-47fa-a68c-51ca10c6a347-kube-api-access-9z8xz\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:58.542425 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:58.542393 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz" Oct 30 22:07:58.542915 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:58.542398 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz" event={"ID":"d3f6086a-cbde-4d85-afab-15e1ad04f832","Type":"ContainerDied","Data":"df4faeee937a430ad091902e3c2099e29e03d25bd4542b2d26af3598dbf4f943"} Oct 30 22:07:58.542915 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:58.542525 5801 scope.go:122] "RemoveContainer" containerID="7d2a8ae067118091ed27baea0c73f8540791ad27ecf0f4f70f7c2a920ec90be3" Oct 30 22:07:58.543878 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:58.543816 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg" event={"ID":"0fc77743-0e00-47fa-a68c-51ca10c6a347","Type":"ContainerDied","Data":"b33f52698cc3ff5260fafd7a09633f586a7ae3aa96edb51c939928256fc01559"} Oct 30 22:07:58.543878 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:58.543831 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg" Oct 30 22:07:58.552985 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:58.552968 5801 scope.go:122] "RemoveContainer" containerID="64ff6c68d44156800c540a216eeb666865817deb6f87f60a7a970e064ebf598a" Oct 30 22:07:58.567256 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:58.567236 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz"] Oct 30 22:07:58.571107 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:58.571086 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0a9006-565445c6c6cwrz"] Oct 30 22:07:58.583276 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:58.583254 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg"] Oct 30 22:07:58.584982 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:58.584963 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a9006-f44c77tkkg"] Oct 30 22:07:59.079443 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:59.079404 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:59.522938 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:59.522906 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fc77743-0e00-47fa-a68c-51ca10c6a347" path="/var/lib/kubelet/pods/0fc77743-0e00-47fa-a68c-51ca10c6a347/volumes" Oct 30 22:07:59.523289 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:07:59.523276 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3f6086a-cbde-4d85-afab-15e1ad04f832" path="/var/lib/kubelet/pods/d3f6086a-cbde-4d85-afab-15e1ad04f832/volumes" Oct 30 22:08:00.278013 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:08:00.277974 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:08:01.399854 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.399821 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc"] Oct 30 22:08:01.400460 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.400190 5801 cpu_manager.go:414] "containerMap: removing container" podUID="a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d" containerName="revoke-keys" Oct 30 22:08:01.400460 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.400201 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d" containerName="revoke-keys" Oct 30 22:08:01.400460 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.400211 5801 cpu_manager.go:414] "containerMap: removing container" podUID="d3f6086a-cbde-4d85-afab-15e1ad04f832" containerName="payload-processing" Oct 30 22:08:01.400460 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.400216 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f6086a-cbde-4d85-afab-15e1ad04f832" containerName="payload-processing" Oct 30 22:08:01.400460 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.400226 5801 cpu_manager.go:414] "containerMap: removing container" podUID="e066ee75-6136-4867-b2b8-40f33b27c8bb" containerName="istio-proxy" Oct 30 22:08:01.400460 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.400231 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e066ee75-6136-4867-b2b8-40f33b27c8bb" containerName="istio-proxy" Oct 30 22:08:01.400460 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.400239 5801 cpu_manager.go:414] "containerMap: removing container" podUID="0fc77743-0e00-47fa-a68c-51ca10c6a347" containerName="payload-pre-processing" Oct 30 22:08:01.400460 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.400244 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc77743-0e00-47fa-a68c-51ca10c6a347" containerName="payload-pre-processing" Oct 30 22:08:01.400460 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.400264 5801 cpu_manager.go:414] "containerMap: removing container" podUID="e066ee75-6136-4867-b2b8-40f33b27c8bb" containerName="istio-proxy" Oct 30 22:08:01.400460 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.400269 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e066ee75-6136-4867-b2b8-40f33b27c8bb" containerName="istio-proxy" Oct 30 22:08:01.400460 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.400356 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e5ce5a-dadb-472c-9fff-f2fa0a06cc2d" containerName="revoke-keys" Oct 30 22:08:01.400460 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.400371 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f6086a-cbde-4d85-afab-15e1ad04f832" containerName="payload-processing" Oct 30 22:08:01.400460 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.400380 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="e066ee75-6136-4867-b2b8-40f33b27c8bb" containerName="istio-proxy" Oct 30 22:08:01.400460 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.400390 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fc77743-0e00-47fa-a68c-51ca10c6a347" containerName="payload-pre-processing" Oct 30 22:08:01.400460 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.400405 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="e066ee75-6136-4867-b2b8-40f33b27c8bb" containerName="istio-proxy" Oct 30 22:08:01.405293 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.405274 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:01.414721 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.414697 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-a86449bf-kserve-self-signed-certs\"" Oct 30 22:08:01.416960 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.416939 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc"] Oct 30 22:08:01.464035 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.464008 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:01.519810 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.519784 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-tls-certs\") pod \"e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc\" (UID: \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\") " pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:01.519937 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.519816 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-dshm\") pod \"e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc\" (UID: \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\") " pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:01.519937 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.519857 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-model-cache\") pod \"e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc\" (UID: \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\") " pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:01.520068 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.519992 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hqxd\" (UniqueName: \"kubernetes.io/projected/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-kube-api-access-7hqxd\") pod \"e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc\" (UID: \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\") " pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:01.520068 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.520039 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-tmp-dir\") pod \"e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc\" (UID: \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\") " pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:01.520176 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.520067 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-home\") pod \"e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc\" (UID: \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\") " pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:01.621561 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.621526 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-tmp-dir\") pod \"e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc\" (UID: \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\") " pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:01.621561 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.621561 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-home\") pod \"e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc\" (UID: \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\") " pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:01.621833 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.621631 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-tls-certs\") pod \"e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc\" (UID: \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\") " pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:01.621833 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.621656 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-dshm\") pod \"e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc\" (UID: \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\") " pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:01.621833 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.621718 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-model-cache\") pod \"e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc\" (UID: \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\") " pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:01.621833 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.621815 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hqxd\" (UniqueName: \"kubernetes.io/projected/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-kube-api-access-7hqxd\") pod \"e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc\" (UID: \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\") " pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:01.622079 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.622009 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-tmp-dir\") pod \"e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc\" (UID: \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\") " pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:01.622079 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.622035 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-home\") pod \"e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc\" (UID: \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\") " pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:01.622170 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.622079 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-model-cache\") pod \"e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc\" (UID: \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\") " pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:01.624030 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.624002 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-dshm\") pod \"e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc\" (UID: \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\") " pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:01.624280 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.624260 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-tls-certs\") pod \"e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc\" (UID: \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\") " pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:01.634170 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.634144 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hqxd\" (UniqueName: \"kubernetes.io/projected/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-kube-api-access-7hqxd\") pod \"e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc\" (UID: \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\") " pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:01.717786 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.717714 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:01.879146 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.879106 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:01.978986 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.978918 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:01.997930 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:01.997905 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc"] Oct 30 22:08:02.000072 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:08:02.000043 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb4fa7ed_5d58_4fae_bef3_b8a42840ed34.slice/crio-e28b1bd904516826e8c4085594145015befb4ce4dc5639474d90d5bcb7898187 WatchSource:0}: Error finding container e28b1bd904516826e8c4085594145015befb4ce4dc5639474d90d5bcb7898187: Status 404 returned error can't find the container with id e28b1bd904516826e8c4085594145015befb4ce4dc5639474d90d5bcb7898187 Oct 30 22:08:02.566375 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:02.566341 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" event={"ID":"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34","Type":"ContainerStarted","Data":"cd0d799e69dee83f897bb9f8002cb55752c6d0f98ad6eaa6598626657a0394b4"} Oct 30 22:08:02.566848 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:02.566385 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" event={"ID":"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34","Type":"ContainerStarted","Data":"e28b1bd904516826e8c4085594145015befb4ce4dc5639474d90d5bcb7898187"} Oct 30 22:08:02.587477 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:02.587436 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" podStartSLOduration=1.587424515 podStartE2EDuration="1.587424515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:02.586840177 +0000 UTC m=+2351.182656867" watchObservedRunningTime="2026-10-30 22:08:02.587424515 +0000 UTC m=+2351.183241175" Oct 30 22:08:02.718537 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:02.718453 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:03.762150 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:03.762115 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:04.673269 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:04.673228 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:06.049778 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:06.047936 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:08:06.049778 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:06.048013 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fm6jd" podUID="913cd5d1-0d49-41ba-921d-603a44b2126a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:06.094775 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:06.094111 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:08:06.094775 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:06.094161 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-hqmk4" podUID="c1e2cfd5-7f5a-4624-a753-30bc982288d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:06.170776 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:06.169837 5801 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:08:06.170776 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:06.169899 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-8fqrl" podUID="57fe57d2-19a5-4ef7-b606-7753b0b7d902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:10.885702 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:10.885663 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:11.718891 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:11.718858 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:11.731253 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:11.731227 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:12.663320 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:12.663293 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:18.264551 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:18.264520 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rxvkm"] Oct 30 22:08:18.267966 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:18.267950 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rxvkm" Oct 30 22:08:18.271771 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:18.271741 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:08:18.272877 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:18.272859 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:08:18.272993 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:18.272879 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 22:08:18.272993 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:18.272904 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:08:18.279164 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:18.279140 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rxvkm"] Oct 30 22:08:18.447282 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:18.447246 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq5tz\" (UniqueName: \"kubernetes.io/projected/0f4fb010-f669-42d4-a654-91ca72cc9e82-kube-api-access-bq5tz\") pod \"maas-api-revoke-sub-452b6d7b-rxvkm\" (UID: \"0f4fb010-f669-42d4-a654-91ca72cc9e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rxvkm" Oct 30 22:08:18.447494 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:18.447430 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4fb010-f669-42d4-a654-91ca72cc9e82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-rxvkm\" (UID: \"0f4fb010-f669-42d4-a654-91ca72cc9e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rxvkm" Oct 30 22:08:18.447494 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:18.447477 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4fb010-f669-42d4-a654-91ca72cc9e82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-rxvkm\" (UID: \"0f4fb010-f669-42d4-a654-91ca72cc9e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rxvkm" Oct 30 22:08:18.548952 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:18.548861 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4fb010-f669-42d4-a654-91ca72cc9e82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-rxvkm\" (UID: \"0f4fb010-f669-42d4-a654-91ca72cc9e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rxvkm" Oct 30 22:08:18.548952 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:18.548924 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4fb010-f669-42d4-a654-91ca72cc9e82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-rxvkm\" (UID: \"0f4fb010-f669-42d4-a654-91ca72cc9e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rxvkm" Oct 30 22:08:18.549194 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:18.548989 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq5tz\" (UniqueName: \"kubernetes.io/projected/0f4fb010-f669-42d4-a654-91ca72cc9e82-kube-api-access-bq5tz\") pod \"maas-api-revoke-sub-452b6d7b-rxvkm\" (UID: \"0f4fb010-f669-42d4-a654-91ca72cc9e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rxvkm" Oct 30 22:08:18.549724 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:18.549690 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4fb010-f669-42d4-a654-91ca72cc9e82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-rxvkm\" (UID: \"0f4fb010-f669-42d4-a654-91ca72cc9e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rxvkm" Oct 30 22:08:18.551265 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:18.551246 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4fb010-f669-42d4-a654-91ca72cc9e82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-rxvkm\" (UID: \"0f4fb010-f669-42d4-a654-91ca72cc9e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rxvkm" Oct 30 22:08:18.561330 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:18.561310 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq5tz\" (UniqueName: \"kubernetes.io/projected/0f4fb010-f669-42d4-a654-91ca72cc9e82-kube-api-access-bq5tz\") pod \"maas-api-revoke-sub-452b6d7b-rxvkm\" (UID: \"0f4fb010-f669-42d4-a654-91ca72cc9e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rxvkm" Oct 30 22:08:18.583164 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:18.583142 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rxvkm" Oct 30 22:08:18.859090 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:18.859062 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rxvkm"] Oct 30 22:08:18.861022 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:08:18.860992 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4fb010_f669_42d4_a654_91ca72cc9e82.slice/crio-724e7eb0495cac3563b7dd531ce108f9f6da16ef986fba115333d042be29091d WatchSource:0}: Error finding container 724e7eb0495cac3563b7dd531ce108f9f6da16ef986fba115333d042be29091d: Status 404 returned error can't find the container with id 724e7eb0495cac3563b7dd531ce108f9f6da16ef986fba115333d042be29091d Oct 30 22:08:19.689612 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:19.689574 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rxvkm" event={"ID":"0f4fb010-f669-42d4-a654-91ca72cc9e82","Type":"ContainerStarted","Data":"eee5b63fc90eefb91cb9f0480fe34fc1ba5a5d639f8f8567e55dec6526c3e037"} Oct 30 22:08:19.690117 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:19.689636 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rxvkm" event={"ID":"0f4fb010-f669-42d4-a654-91ca72cc9e82","Type":"ContainerStarted","Data":"724e7eb0495cac3563b7dd531ce108f9f6da16ef986fba115333d042be29091d"} Oct 30 22:08:19.712483 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:19.712437 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rxvkm" podStartSLOduration=1.712424021 podStartE2EDuration="1.712424021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:19.711369126 +0000 UTC m=+2368.307185806" watchObservedRunningTime="2026-10-30 22:08:19.712424021 +0000 UTC m=+2368.308240677" Oct 30 22:08:20.699904 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:20.699860 5801 generic.go:356] "Generic (PLEG): container finished" podID="0f4fb010-f669-42d4-a654-91ca72cc9e82" containerID="eee5b63fc90eefb91cb9f0480fe34fc1ba5a5d639f8f8567e55dec6526c3e037" exitCode=0 Oct 30 22:08:20.700437 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:20.699922 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rxvkm" event={"ID":"0f4fb010-f669-42d4-a654-91ca72cc9e82","Type":"ContainerDied","Data":"eee5b63fc90eefb91cb9f0480fe34fc1ba5a5d639f8f8567e55dec6526c3e037"} Oct 30 22:08:22.691775 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:22.690846 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rxvkm" Oct 30 22:08:22.738779 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:22.732225 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f4fb010-f669-42d4-a654-91ca72cc9e82-kube-api-access-bq5tz\" (UniqueName: \"kubernetes.io/projected/0f4fb010-f669-42d4-a654-91ca72cc9e82-kube-api-access-bq5tz\") pod \"0f4fb010-f669-42d4-a654-91ca72cc9e82\" (UID: \"0f4fb010-f669-42d4-a654-91ca72cc9e82\") " Oct 30 22:08:22.742556 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:22.742431 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4fb010-f669-42d4-a654-91ca72cc9e82-kube-api-access-bq5tz" pod "0f4fb010-f669-42d4-a654-91ca72cc9e82" (UID: "0f4fb010-f669-42d4-a654-91ca72cc9e82"). InnerVolumeSpecName "kube-api-access-bq5tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:22.823981 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:22.823944 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rxvkm" event={"ID":"0f4fb010-f669-42d4-a654-91ca72cc9e82","Type":"ContainerDied","Data":"724e7eb0495cac3563b7dd531ce108f9f6da16ef986fba115333d042be29091d"} Oct 30 22:08:22.830844 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:22.830816 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="724e7eb0495cac3563b7dd531ce108f9f6da16ef986fba115333d042be29091d" Oct 30 22:08:22.840772 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:22.831067 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rxvkm" Oct 30 22:08:22.885789 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:22.853125 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f4fb010-f669-42d4-a654-91ca72cc9e82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4fb010-f669-42d4-a654-91ca72cc9e82-maas-api-xks-ca\") pod \"0f4fb010-f669-42d4-a654-91ca72cc9e82\" (UID: \"0f4fb010-f669-42d4-a654-91ca72cc9e82\") " Oct 30 22:08:22.886027 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:22.886004 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f4fb010-f669-42d4-a654-91ca72cc9e82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4fb010-f669-42d4-a654-91ca72cc9e82-maas-api-openshift-ca\") pod \"0f4fb010-f669-42d4-a654-91ca72cc9e82\" (UID: \"0f4fb010-f669-42d4-a654-91ca72cc9e82\") " Oct 30 22:08:22.899771 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:22.886458 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq5tz\" (UniqueName: \"kubernetes.io/projected/0f4fb010-f669-42d4-a654-91ca72cc9e82-kube-api-access-bq5tz\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:22.900273 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:22.900247 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f4fb010-f669-42d4-a654-91ca72cc9e82-maas-api-openshift-ca" pod "0f4fb010-f669-42d4-a654-91ca72cc9e82" (UID: "0f4fb010-f669-42d4-a654-91ca72cc9e82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:22.910213 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:22.910189 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f4fb010-f669-42d4-a654-91ca72cc9e82-maas-api-xks-ca" pod "0f4fb010-f669-42d4-a654-91ca72cc9e82" (UID: "0f4fb010-f669-42d4-a654-91ca72cc9e82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:23.001515 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:23.001483 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4fb010-f669-42d4-a654-91ca72cc9e82-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:23.001806 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:23.001784 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4fb010-f669-42d4-a654-91ca72cc9e82-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:29.281581 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:29.281540 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:30.277341 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:08:30.277309 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:08:30.658260 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:30.658226 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc"] Oct 30 22:08:30.724809 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:30.724778 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:33.021257 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:33.021205 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:34.419915 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:34.419881 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c8jdn"] Oct 30 22:08:34.420331 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:34.420298 5801 cpu_manager.go:414] "containerMap: removing container" podUID="0f4fb010-f669-42d4-a654-91ca72cc9e82" containerName="revoke-keys" Oct 30 22:08:34.420331 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:34.420311 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f4fb010-f669-42d4-a654-91ca72cc9e82" containerName="revoke-keys" Oct 30 22:08:34.420420 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:34.420397 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f4fb010-f669-42d4-a654-91ca72cc9e82" containerName="revoke-keys" Oct 30 22:08:34.423626 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:34.423611 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c8jdn" Oct 30 22:08:34.431438 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:34.431414 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:08:34.431585 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:34.431460 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:08:34.431585 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:34.431546 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:08:34.431585 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:34.431570 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 22:08:34.434943 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:34.434921 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c8jdn"] Oct 30 22:08:34.498051 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:34.498013 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1124264-b8d0-4bde-b039-d3f9f68d5802-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-c8jdn\" (UID: \"f1124264-b8d0-4bde-b039-d3f9f68d5802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c8jdn" Oct 30 22:08:34.498216 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:34.498123 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4r5p\" (UniqueName: \"kubernetes.io/projected/f1124264-b8d0-4bde-b039-d3f9f68d5802-kube-api-access-x4r5p\") pod \"maas-api-revoke-sub-3e31a506-c8jdn\" (UID: \"f1124264-b8d0-4bde-b039-d3f9f68d5802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c8jdn" Oct 30 22:08:34.498216 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:34.498154 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1124264-b8d0-4bde-b039-d3f9f68d5802-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-c8jdn\" (UID: \"f1124264-b8d0-4bde-b039-d3f9f68d5802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c8jdn" Oct 30 22:08:34.599702 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:34.599661 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1124264-b8d0-4bde-b039-d3f9f68d5802-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-c8jdn\" (UID: \"f1124264-b8d0-4bde-b039-d3f9f68d5802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c8jdn" Oct 30 22:08:34.599933 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:34.599790 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4r5p\" (UniqueName: \"kubernetes.io/projected/f1124264-b8d0-4bde-b039-d3f9f68d5802-kube-api-access-x4r5p\") pod \"maas-api-revoke-sub-3e31a506-c8jdn\" (UID: \"f1124264-b8d0-4bde-b039-d3f9f68d5802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c8jdn" Oct 30 22:08:34.599933 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:34.599835 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1124264-b8d0-4bde-b039-d3f9f68d5802-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-c8jdn\" (UID: \"f1124264-b8d0-4bde-b039-d3f9f68d5802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c8jdn" Oct 30 22:08:34.600411 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:34.600381 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1124264-b8d0-4bde-b039-d3f9f68d5802-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-c8jdn\" (UID: \"f1124264-b8d0-4bde-b039-d3f9f68d5802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c8jdn" Oct 30 22:08:34.602156 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:34.602129 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1124264-b8d0-4bde-b039-d3f9f68d5802-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-c8jdn\" (UID: \"f1124264-b8d0-4bde-b039-d3f9f68d5802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c8jdn" Oct 30 22:08:34.612894 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:34.612868 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4r5p\" (UniqueName: \"kubernetes.io/projected/f1124264-b8d0-4bde-b039-d3f9f68d5802-kube-api-access-x4r5p\") pod \"maas-api-revoke-sub-3e31a506-c8jdn\" (UID: \"f1124264-b8d0-4bde-b039-d3f9f68d5802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c8jdn" Oct 30 22:08:34.738269 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:34.738180 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c8jdn" Oct 30 22:08:35.026632 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:35.026537 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c8jdn"] Oct 30 22:08:35.031313 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:08:35.031282 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1124264_b8d0_4bde_b039_d3f9f68d5802.slice/crio-5f2be09382e1a7262ae8d82099733a168e16513f8d045057da285e9a886f45b7 WatchSource:0}: Error finding container 5f2be09382e1a7262ae8d82099733a168e16513f8d045057da285e9a886f45b7: Status 404 returned error can't find the container with id 5f2be09382e1a7262ae8d82099733a168e16513f8d045057da285e9a886f45b7 Oct 30 22:08:35.953658 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:35.953619 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c8jdn" event={"ID":"f1124264-b8d0-4bde-b039-d3f9f68d5802","Type":"ContainerStarted","Data":"fd7f277e4a0a587f788e966145e69f47f597723645468abbe5fe72acc3c0b5c7"} Oct 30 22:08:35.959930 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:35.959897 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c8jdn" event={"ID":"f1124264-b8d0-4bde-b039-d3f9f68d5802","Type":"ContainerStarted","Data":"5f2be09382e1a7262ae8d82099733a168e16513f8d045057da285e9a886f45b7"} Oct 30 22:08:36.982385 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:36.982345 5801 generic.go:356] "Generic (PLEG): container finished" podID="f1124264-b8d0-4bde-b039-d3f9f68d5802" containerID="fd7f277e4a0a587f788e966145e69f47f597723645468abbe5fe72acc3c0b5c7" exitCode=0 Oct 30 22:08:36.983039 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:36.983014 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c8jdn" event={"ID":"f1124264-b8d0-4bde-b039-d3f9f68d5802","Type":"ContainerDied","Data":"fd7f277e4a0a587f788e966145e69f47f597723645468abbe5fe72acc3c0b5c7"} Oct 30 22:08:37.039051 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:37.039019 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38e5507c-n8zhl"] Oct 30 22:08:37.043227 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:37.043190 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38e5507c-n8zhl"] Oct 30 22:08:37.525500 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:37.525465 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba6a329-a55c-4567-a09d-194f79b2b09b" path="/var/lib/kubelet/pods/5ba6a329-a55c-4567-a09d-194f79b2b09b/volumes" Oct 30 22:08:38.140154 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:38.140131 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c8jdn" Oct 30 22:08:38.156630 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:38.156599 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1124264-b8d0-4bde-b039-d3f9f68d5802-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1124264-b8d0-4bde-b039-d3f9f68d5802-maas-api-xks-ca\") pod \"f1124264-b8d0-4bde-b039-d3f9f68d5802\" (UID: \"f1124264-b8d0-4bde-b039-d3f9f68d5802\") " Oct 30 22:08:38.156773 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:38.156672 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1124264-b8d0-4bde-b039-d3f9f68d5802-kube-api-access-x4r5p\" (UniqueName: \"kubernetes.io/projected/f1124264-b8d0-4bde-b039-d3f9f68d5802-kube-api-access-x4r5p\") pod \"f1124264-b8d0-4bde-b039-d3f9f68d5802\" (UID: \"f1124264-b8d0-4bde-b039-d3f9f68d5802\") " Oct 30 22:08:38.156773 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:38.156713 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1124264-b8d0-4bde-b039-d3f9f68d5802-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1124264-b8d0-4bde-b039-d3f9f68d5802-maas-api-openshift-ca\") pod \"f1124264-b8d0-4bde-b039-d3f9f68d5802\" (UID: \"f1124264-b8d0-4bde-b039-d3f9f68d5802\") " Oct 30 22:08:38.157048 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:38.157026 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1124264-b8d0-4bde-b039-d3f9f68d5802-maas-api-openshift-ca" pod "f1124264-b8d0-4bde-b039-d3f9f68d5802" (UID: "f1124264-b8d0-4bde-b039-d3f9f68d5802"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:38.157194 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:38.157179 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1124264-b8d0-4bde-b039-d3f9f68d5802-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:38.158594 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:38.158570 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1124264-b8d0-4bde-b039-d3f9f68d5802-kube-api-access-x4r5p" pod "f1124264-b8d0-4bde-b039-d3f9f68d5802" (UID: "f1124264-b8d0-4bde-b039-d3f9f68d5802"). InnerVolumeSpecName "kube-api-access-x4r5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:38.158716 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:38.158648 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1124264-b8d0-4bde-b039-d3f9f68d5802-maas-api-xks-ca" pod "f1124264-b8d0-4bde-b039-d3f9f68d5802" (UID: "f1124264-b8d0-4bde-b039-d3f9f68d5802"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:38.258523 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:38.258458 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1124264-b8d0-4bde-b039-d3f9f68d5802-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:38.258523 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:38.258485 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4r5p\" (UniqueName: \"kubernetes.io/projected/f1124264-b8d0-4bde-b039-d3f9f68d5802-kube-api-access-x4r5p\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:39.002217 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:39.002187 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c8jdn" Oct 30 22:08:39.002217 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:39.002196 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c8jdn" event={"ID":"f1124264-b8d0-4bde-b039-d3f9f68d5802","Type":"ContainerDied","Data":"5f2be09382e1a7262ae8d82099733a168e16513f8d045057da285e9a886f45b7"} Oct 30 22:08:39.002439 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:39.002238 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f2be09382e1a7262ae8d82099733a168e16513f8d045057da285e9a886f45b7" Oct 30 22:08:45.678871 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:45.678829 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" podUID="eb4fa7ed-5d58-4fae-bef3-b8a42840ed34" containerName="main" containerID="cri-o://cd0d799e69dee83f897bb9f8002cb55752c6d0f98ad6eaa6598626657a0394b4" gracePeriod=45 Oct 30 22:08:46.137777 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:46.136932 5801 generic.go:356] "Generic (PLEG): container finished" podID="eb4fa7ed-5d58-4fae-bef3-b8a42840ed34" containerID="cd0d799e69dee83f897bb9f8002cb55752c6d0f98ad6eaa6598626657a0394b4" exitCode=0 Oct 30 22:08:46.137777 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:46.137157 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" event={"ID":"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34","Type":"ContainerDied","Data":"cd0d799e69dee83f897bb9f8002cb55752c6d0f98ad6eaa6598626657a0394b4"} Oct 30 22:08:46.427775 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:46.426912 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:46.494220 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:46.481009 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-kube-api-access-7hqxd\" (UniqueName: \"kubernetes.io/projected/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-kube-api-access-7hqxd\") pod \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\" (UID: \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\") " Oct 30 22:08:46.494220 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:46.481086 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-tls-certs\") pod \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\" (UID: \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\") " Oct 30 22:08:46.494220 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:46.481128 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-model-cache\") pod \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\" (UID: \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\") " Oct 30 22:08:46.494220 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:46.481166 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-dshm\") pod \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\" (UID: \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\") " Oct 30 22:08:46.494220 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:46.481226 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-tmp-dir\") pod \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\" (UID: \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\") " Oct 30 22:08:46.494220 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:46.481327 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-home\" (UniqueName: \"kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-home\") pod \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\" (UID: \"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34\") " Oct 30 22:08:46.495099 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:46.495039 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-tmp-dir" pod "eb4fa7ed-5d58-4fae-bef3-b8a42840ed34" (UID: "eb4fa7ed-5d58-4fae-bef3-b8a42840ed34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:46.495355 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:46.495303 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-model-cache" pod "eb4fa7ed-5d58-4fae-bef3-b8a42840ed34" (UID: "eb4fa7ed-5d58-4fae-bef3-b8a42840ed34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:46.544563 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:46.523837 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-home" pod "eb4fa7ed-5d58-4fae-bef3-b8a42840ed34" (UID: "eb4fa7ed-5d58-4fae-bef3-b8a42840ed34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:46.550907 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:46.550864 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-kube-api-access-7hqxd" pod "eb4fa7ed-5d58-4fae-bef3-b8a42840ed34" (UID: "eb4fa7ed-5d58-4fae-bef3-b8a42840ed34"). InnerVolumeSpecName "kube-api-access-7hqxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:46.551177 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:46.551145 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-dshm" pod "eb4fa7ed-5d58-4fae-bef3-b8a42840ed34" (UID: "eb4fa7ed-5d58-4fae-bef3-b8a42840ed34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:46.559835 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:46.558856 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-tls-certs" pod "eb4fa7ed-5d58-4fae-bef3-b8a42840ed34" (UID: "eb4fa7ed-5d58-4fae-bef3-b8a42840ed34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:46.588810 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:46.587317 5801 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:46.588810 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:46.587367 5801 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:46.588810 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:46.587383 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hqxd\" (UniqueName: \"kubernetes.io/projected/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-kube-api-access-7hqxd\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:46.588810 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:46.587399 5801 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:46.588810 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:46.587415 5801 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:46.588810 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:46.587430 5801 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:47.153316 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:47.153275 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" event={"ID":"eb4fa7ed-5d58-4fae-bef3-b8a42840ed34","Type":"ContainerDied","Data":"e28b1bd904516826e8c4085594145015befb4ce4dc5639474d90d5bcb7898187"} Oct 30 22:08:47.157783 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:47.153894 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc" Oct 30 22:08:47.162517 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:47.154241 5801 scope.go:122] "RemoveContainer" containerID="cd0d799e69dee83f897bb9f8002cb55752c6d0f98ad6eaa6598626657a0394b4" Oct 30 22:08:47.202500 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:47.202462 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc"] Oct 30 22:08:47.214891 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:47.214868 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-a86449bf-kserve-74d54fd65-pnvqc"] Oct 30 22:08:47.563939 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:47.563906 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb4fa7ed-5d58-4fae-bef3-b8a42840ed34" path="/var/lib/kubelet/pods/eb4fa7ed-5d58-4fae-bef3-b8a42840ed34/volumes" Oct 30 22:08:55.149735 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:55.149699 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0fa3aa-5ktnz"] Oct 30 22:08:55.150113 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:55.150062 5801 cpu_manager.go:414] "containerMap: removing container" podUID="f1124264-b8d0-4bde-b039-d3f9f68d5802" containerName="revoke-keys" Oct 30 22:08:55.150113 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:55.150075 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1124264-b8d0-4bde-b039-d3f9f68d5802" containerName="revoke-keys" Oct 30 22:08:55.150113 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:55.150091 5801 cpu_manager.go:414] "containerMap: removing container" podUID="eb4fa7ed-5d58-4fae-bef3-b8a42840ed34" containerName="main" Oct 30 22:08:55.150113 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:55.150096 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb4fa7ed-5d58-4fae-bef3-b8a42840ed34" containerName="main" Oct 30 22:08:55.150272 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:55.150185 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1124264-b8d0-4bde-b039-d3f9f68d5802" containerName="revoke-keys" Oct 30 22:08:55.150272 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:55.150194 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb4fa7ed-5d58-4fae-bef3-b8a42840ed34" containerName="main" Oct 30 22:08:55.153202 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:55.153186 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0fa3aa-5ktnz" Oct 30 22:08:55.159698 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:55.159673 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:08:55.166294 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:55.166274 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:08:55.166453 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:55.166340 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 22:08:55.166535 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:55.166458 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0fa3aa-5ktnz"] Oct 30 22:08:55.225599 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:55.225562 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137ef9a5-5d59-4c38-8890-fe71531e4d80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db0fa3aa-5ktnz\" (UID: \"137ef9a5-5d59-4c38-8890-fe71531e4d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0fa3aa-5ktnz" Oct 30 22:08:55.225810 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:55.225689 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjt75\" (UniqueName: \"kubernetes.io/projected/137ef9a5-5d59-4c38-8890-fe71531e4d80-kube-api-access-fjt75\") pod \"maas-api-revoke-sub-db0fa3aa-5ktnz\" (UID: \"137ef9a5-5d59-4c38-8890-fe71531e4d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0fa3aa-5ktnz" Oct 30 22:08:55.225810 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:55.225776 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137ef9a5-5d59-4c38-8890-fe71531e4d80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db0fa3aa-5ktnz\" (UID: \"137ef9a5-5d59-4c38-8890-fe71531e4d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0fa3aa-5ktnz" Oct 30 22:08:55.259942 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:55.259916 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:08:55.327084 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:55.327058 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137ef9a5-5d59-4c38-8890-fe71531e4d80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db0fa3aa-5ktnz\" (UID: \"137ef9a5-5d59-4c38-8890-fe71531e4d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0fa3aa-5ktnz" Oct 30 22:08:55.327238 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:55.327123 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137ef9a5-5d59-4c38-8890-fe71531e4d80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db0fa3aa-5ktnz\" (UID: \"137ef9a5-5d59-4c38-8890-fe71531e4d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0fa3aa-5ktnz" Oct 30 22:08:55.327238 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:55.327180 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjt75\" (UniqueName: \"kubernetes.io/projected/137ef9a5-5d59-4c38-8890-fe71531e4d80-kube-api-access-fjt75\") pod \"maas-api-revoke-sub-db0fa3aa-5ktnz\" (UID: \"137ef9a5-5d59-4c38-8890-fe71531e4d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0fa3aa-5ktnz" Oct 30 22:08:55.327707 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:55.327682 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137ef9a5-5d59-4c38-8890-fe71531e4d80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db0fa3aa-5ktnz\" (UID: \"137ef9a5-5d59-4c38-8890-fe71531e4d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0fa3aa-5ktnz" Oct 30 22:08:55.329346 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:55.329326 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137ef9a5-5d59-4c38-8890-fe71531e4d80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db0fa3aa-5ktnz\" (UID: \"137ef9a5-5d59-4c38-8890-fe71531e4d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0fa3aa-5ktnz" Oct 30 22:08:55.339567 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:55.339546 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjt75\" (UniqueName: \"kubernetes.io/projected/137ef9a5-5d59-4c38-8890-fe71531e4d80-kube-api-access-fjt75\") pod \"maas-api-revoke-sub-db0fa3aa-5ktnz\" (UID: \"137ef9a5-5d59-4c38-8890-fe71531e4d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0fa3aa-5ktnz" Oct 30 22:08:55.467579 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:55.467511 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0fa3aa-5ktnz" Oct 30 22:08:55.746244 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:55.746220 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0fa3aa-5ktnz"] Oct 30 22:08:55.748027 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:08:55.748000 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137ef9a5_5d59_4c38_8890_fe71531e4d80.slice/crio-c076ace11c4950ab6504ac114cde81ce0347a8582a82ac56941cba3e6541f7a4 WatchSource:0}: Error finding container c076ace11c4950ab6504ac114cde81ce0347a8582a82ac56941cba3e6541f7a4: Status 404 returned error can't find the container with id c076ace11c4950ab6504ac114cde81ce0347a8582a82ac56941cba3e6541f7a4 Oct 30 22:08:56.233161 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:56.233133 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0fa3aa-5ktnz" event={"ID":"137ef9a5-5d59-4c38-8890-fe71531e4d80","Type":"ContainerStarted","Data":"3e8ec8f72ce6f2f57bab4f563e4af478ccef48c55010a46e8a02e6a014831d20"} Oct 30 22:08:56.233513 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:56.233182 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0fa3aa-5ktnz" event={"ID":"137ef9a5-5d59-4c38-8890-fe71531e4d80","Type":"ContainerStarted","Data":"c076ace11c4950ab6504ac114cde81ce0347a8582a82ac56941cba3e6541f7a4"} Oct 30 22:08:56.262393 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:56.262346 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0fa3aa-5ktnz" podStartSLOduration=1.262332782 podStartE2EDuration="1.262332782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:56.26058016 +0000 UTC m=+2404.856396822" watchObservedRunningTime="2026-10-30 22:08:56.262332782 +0000 UTC m=+2404.858149457" Oct 30 22:08:57.239555 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:57.239516 5801 generic.go:356] "Generic (PLEG): container finished" podID="137ef9a5-5d59-4c38-8890-fe71531e4d80" containerID="3e8ec8f72ce6f2f57bab4f563e4af478ccef48c55010a46e8a02e6a014831d20" exitCode=0 Oct 30 22:08:57.240000 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:57.239602 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0fa3aa-5ktnz" event={"ID":"137ef9a5-5d59-4c38-8890-fe71531e4d80","Type":"ContainerDied","Data":"3e8ec8f72ce6f2f57bab4f563e4af478ccef48c55010a46e8a02e6a014831d20"} Oct 30 22:08:58.385367 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:58.385333 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0fa3aa-5ktnz" Oct 30 22:08:58.456308 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:58.456282 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/137ef9a5-5d59-4c38-8890-fe71531e4d80-kube-api-access-fjt75\" (UniqueName: \"kubernetes.io/projected/137ef9a5-5d59-4c38-8890-fe71531e4d80-kube-api-access-fjt75\") pod \"137ef9a5-5d59-4c38-8890-fe71531e4d80\" (UID: \"137ef9a5-5d59-4c38-8890-fe71531e4d80\") " Oct 30 22:08:58.456459 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:58.456337 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/137ef9a5-5d59-4c38-8890-fe71531e4d80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137ef9a5-5d59-4c38-8890-fe71531e4d80-maas-api-xks-ca\") pod \"137ef9a5-5d59-4c38-8890-fe71531e4d80\" (UID: \"137ef9a5-5d59-4c38-8890-fe71531e4d80\") " Oct 30 22:08:58.456459 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:58.456410 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/137ef9a5-5d59-4c38-8890-fe71531e4d80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137ef9a5-5d59-4c38-8890-fe71531e4d80-maas-api-openshift-ca\") pod \"137ef9a5-5d59-4c38-8890-fe71531e4d80\" (UID: \"137ef9a5-5d59-4c38-8890-fe71531e4d80\") " Oct 30 22:08:58.456805 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:58.456771 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137ef9a5-5d59-4c38-8890-fe71531e4d80-maas-api-openshift-ca" pod "137ef9a5-5d59-4c38-8890-fe71531e4d80" (UID: "137ef9a5-5d59-4c38-8890-fe71531e4d80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:58.458455 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:58.458433 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137ef9a5-5d59-4c38-8890-fe71531e4d80-kube-api-access-fjt75" pod "137ef9a5-5d59-4c38-8890-fe71531e4d80" (UID: "137ef9a5-5d59-4c38-8890-fe71531e4d80"). InnerVolumeSpecName "kube-api-access-fjt75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:58.458551 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:58.458443 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137ef9a5-5d59-4c38-8890-fe71531e4d80-maas-api-xks-ca" pod "137ef9a5-5d59-4c38-8890-fe71531e4d80" (UID: "137ef9a5-5d59-4c38-8890-fe71531e4d80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:58.558001 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:58.557979 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjt75\" (UniqueName: \"kubernetes.io/projected/137ef9a5-5d59-4c38-8890-fe71531e4d80-kube-api-access-fjt75\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:58.558001 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:58.558001 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137ef9a5-5d59-4c38-8890-fe71531e4d80-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:58.558155 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:58.558012 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137ef9a5-5d59-4c38-8890-fe71531e4d80-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:59.215860 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:59.215824 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:59.251238 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:59.251213 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0fa3aa-5ktnz" Oct 30 22:08:59.251393 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:59.251250 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0fa3aa-5ktnz" event={"ID":"137ef9a5-5d59-4c38-8890-fe71531e4d80","Type":"ContainerDied","Data":"c076ace11c4950ab6504ac114cde81ce0347a8582a82ac56941cba3e6541f7a4"} Oct 30 22:08:59.251393 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:59.251303 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c076ace11c4950ab6504ac114cde81ce0347a8582a82ac56941cba3e6541f7a4" Oct 30 22:08:59.431959 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:08:59.431932 5801 scope.go:122] "RemoveContainer" containerID="6317d618ff4defba2f2e6e7a023bf8b066cee8ccd5d84ab1793d8d6ef00f97bf" Oct 30 22:09:00.277201 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:09:00.277167 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:09:01.085256 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:01.085220 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:07.028904 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:07.028866 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vt7gs"] Oct 30 22:09:07.033841 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:07.033811 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vt7gs"] Oct 30 22:09:07.522801 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:07.522746 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d90664e3-47d3-4247-8804-10f1f7d3c09d" path="/var/lib/kubelet/pods/d90664e3-47d3-4247-8804-10f1f7d3c09d/volumes" Oct 30 22:09:14.785139 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:14.785096 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:14.931255 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:14.931224 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:15.245487 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.245455 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt"] Oct 30 22:09:15.245891 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.245877 5801 cpu_manager.go:414] "containerMap: removing container" podUID="137ef9a5-5d59-4c38-8890-fe71531e4d80" containerName="revoke-keys" Oct 30 22:09:15.245946 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.245893 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137ef9a5-5d59-4c38-8890-fe71531e4d80" containerName="revoke-keys" Oct 30 22:09:15.245996 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.245961 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="137ef9a5-5d59-4c38-8890-fe71531e4d80" containerName="revoke-keys" Oct 30 22:09:15.250536 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.250516 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.271549 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.271520 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-83f27219-openshift-default-dockercfg-6l2s8\"" Oct 30 22:09:15.272944 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.272915 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt"] Oct 30 22:09:15.406405 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.406377 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hngxw\" (UniqueName: \"kubernetes.io/projected/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-kube-api-access-hngxw\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.406631 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.406428 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-token\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.406631 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.406452 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-credential-socket\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.406631 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.406545 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-podinfo\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.406631 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.406583 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-workload-certs\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.406809 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.406656 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-workload-socket\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.406809 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.406697 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-envoy\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.406809 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.406717 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istiod-ca-cert\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.406809 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.406791 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-data\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.462452 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.462416 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:15.507796 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.507682 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hngxw\" (UniqueName: \"kubernetes.io/projected/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-kube-api-access-hngxw\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.507796 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.507774 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-token\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.508046 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.507801 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-credential-socket\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.508046 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.507823 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-podinfo\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.508046 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.507856 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-workload-certs\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.508046 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.507911 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-workload-socket\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.508046 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.507941 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-envoy\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.508046 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.507989 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istiod-ca-cert\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.508046 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.508047 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-data\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.508432 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.508267 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-credential-socket\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.508432 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.508316 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-workload-certs\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.508432 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.508353 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-workload-socket\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.508432 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.508424 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-data\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.508689 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.508670 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istiod-ca-cert\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.510183 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.510154 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-envoy\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.510531 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.510500 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-podinfo\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.529527 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.529502 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hngxw\" (UniqueName: \"kubernetes.io/projected/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-kube-api-access-hngxw\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.538377 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.538353 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-token\") pod \"e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.563360 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.563337 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:15.666766 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.666708 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:15.874675 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.874645 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt"] Oct 30 22:09:15.878287 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:09:15.878259 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf781d69c_c4c6_4323_b8e3_28fccbf1b22f.slice/crio-72356c848f262dcbe357f3153bd0ab18ed7d3ee0262e5265bf62689c31c3e69d WatchSource:0}: Error finding container 72356c848f262dcbe357f3153bd0ab18ed7d3ee0262e5265bf62689c31c3e69d: Status 404 returned error can't find the container with id 72356c848f262dcbe357f3153bd0ab18ed7d3ee0262e5265bf62689c31c3e69d Oct 30 22:09:15.880383 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.880351 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:09:15.880479 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.880429 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:09:15.880479 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:15.880474 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:09:16.338682 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:16.338649 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" event={"ID":"f781d69c-c4c6-4323-b8e3-28fccbf1b22f","Type":"ContainerStarted","Data":"a55958e6adf52d2bec2098d46884922c5bfefef2751af60a1e4ffc3527cc1b52"} Oct 30 22:09:16.338886 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:16.338702 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" event={"ID":"f781d69c-c4c6-4323-b8e3-28fccbf1b22f","Type":"ContainerStarted","Data":"72356c848f262dcbe357f3153bd0ab18ed7d3ee0262e5265bf62689c31c3e69d"} Oct 30 22:09:16.393325 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:16.389766 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" podStartSLOduration=1.389734841 podStartE2EDuration="1.389734841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:16.386839356 +0000 UTC m=+2424.982656014" watchObservedRunningTime="2026-10-30 22:09:16.389734841 +0000 UTC m=+2424.985551502" Oct 30 22:09:16.563956 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:16.563917 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:16.568863 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:16.568834 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:17.343487 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:17.343455 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:17.344518 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:17.344499 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:17.958912 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:17.958879 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:18.077581 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:18.077542 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:19.224894 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.224857 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:19.258586 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.258556 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv"] Oct 30 22:09:19.261946 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.261930 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv" Oct 30 22:09:19.272686 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.272667 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-83f27219\"" Oct 30 22:09:19.280871 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.280849 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv"] Oct 30 22:09:19.310804 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.310745 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52"] Oct 30 22:09:19.314295 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.314279 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52" Oct 30 22:09:19.321507 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.321487 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-83f27219-dockercfg-2jrbs\"" Oct 30 22:09:19.327466 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.327444 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52"] Oct 30 22:09:19.375871 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.375839 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:19.446035 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.445981 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9138369d-c863-4773-b3d6-4060fa6f16f4-plugins-config-volume\") pod \"payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52\" (UID: \"9138369d-c863-4773-b3d6-4060fa6f16f4\") " pod="openshift-ingress/payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52" Oct 30 22:09:19.446257 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.446114 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2bzq\" (UniqueName: \"kubernetes.io/projected/9138369d-c863-4773-b3d6-4060fa6f16f4-kube-api-access-z2bzq\") pod \"payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52\" (UID: \"9138369d-c863-4773-b3d6-4060fa6f16f4\") " pod="openshift-ingress/payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52" Oct 30 22:09:19.446257 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.446235 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312b05e0-c4aa-410f-b8ea-038f067ba880-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv\" (UID: \"312b05e0-c4aa-410f-b8ea-038f067ba880\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv" Oct 30 22:09:19.446373 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.446268 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmff2\" (UniqueName: \"kubernetes.io/projected/312b05e0-c4aa-410f-b8ea-038f067ba880-kube-api-access-tmff2\") pod \"payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv\" (UID: \"312b05e0-c4aa-410f-b8ea-038f067ba880\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv" Oct 30 22:09:19.547602 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.547525 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9138369d-c863-4773-b3d6-4060fa6f16f4-plugins-config-volume\") pod \"payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52\" (UID: \"9138369d-c863-4773-b3d6-4060fa6f16f4\") " pod="openshift-ingress/payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52" Oct 30 22:09:19.547602 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.547564 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2bzq\" (UniqueName: \"kubernetes.io/projected/9138369d-c863-4773-b3d6-4060fa6f16f4-kube-api-access-z2bzq\") pod \"payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52\" (UID: \"9138369d-c863-4773-b3d6-4060fa6f16f4\") " pod="openshift-ingress/payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52" Oct 30 22:09:19.547843 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.547630 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312b05e0-c4aa-410f-b8ea-038f067ba880-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv\" (UID: \"312b05e0-c4aa-410f-b8ea-038f067ba880\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv" Oct 30 22:09:19.547843 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.547660 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmff2\" (UniqueName: \"kubernetes.io/projected/312b05e0-c4aa-410f-b8ea-038f067ba880-kube-api-access-tmff2\") pod \"payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv\" (UID: \"312b05e0-c4aa-410f-b8ea-038f067ba880\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv" Oct 30 22:09:19.548356 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.548336 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9138369d-c863-4773-b3d6-4060fa6f16f4-plugins-config-volume\") pod \"payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52\" (UID: \"9138369d-c863-4773-b3d6-4060fa6f16f4\") " pod="openshift-ingress/payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52" Oct 30 22:09:19.548402 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.548369 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312b05e0-c4aa-410f-b8ea-038f067ba880-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv\" (UID: \"312b05e0-c4aa-410f-b8ea-038f067ba880\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv" Oct 30 22:09:19.560306 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.560277 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmff2\" (UniqueName: \"kubernetes.io/projected/312b05e0-c4aa-410f-b8ea-038f067ba880-kube-api-access-tmff2\") pod \"payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv\" (UID: \"312b05e0-c4aa-410f-b8ea-038f067ba880\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv" Oct 30 22:09:19.560500 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.560478 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2bzq\" (UniqueName: \"kubernetes.io/projected/9138369d-c863-4773-b3d6-4060fa6f16f4-kube-api-access-z2bzq\") pod \"payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52\" (UID: \"9138369d-c863-4773-b3d6-4060fa6f16f4\") " pod="openshift-ingress/payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52" Oct 30 22:09:19.575298 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.575272 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv" Oct 30 22:09:19.627991 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.627963 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52" Oct 30 22:09:19.860644 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.860606 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv"] Oct 30 22:09:19.863955 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:09:19.863922 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312b05e0_c4aa_410f_b8ea_038f067ba880.slice/crio-5bbcef5489ec6eea4d12ce343a0f833e26c899df3155df9e5b9423138d8d4962 WatchSource:0}: Error finding container 5bbcef5489ec6eea4d12ce343a0f833e26c899df3155df9e5b9423138d8d4962: Status 404 returned error can't find the container with id 5bbcef5489ec6eea4d12ce343a0f833e26c899df3155df9e5b9423138d8d4962 Oct 30 22:09:19.906494 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.906463 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:19.920569 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:19.920450 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52"] Oct 30 22:09:19.922661 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:09:19.922636 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9138369d_c863_4773_b3d6_4060fa6f16f4.slice/crio-bc13eb90602a727dd4f1e59ab6e8a52ce52499f3e08bee0335666d3050b9d144 WatchSource:0}: Error finding container bc13eb90602a727dd4f1e59ab6e8a52ce52499f3e08bee0335666d3050b9d144: Status 404 returned error can't find the container with id bc13eb90602a727dd4f1e59ab6e8a52ce52499f3e08bee0335666d3050b9d144 Oct 30 22:09:20.019932 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:20.019887 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:20.360643 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:20.360601 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv" event={"ID":"312b05e0-c4aa-410f-b8ea-038f067ba880","Type":"ContainerStarted","Data":"5ecf9c084c9414fa9c8b53f18dc2b539dff6866ae5fd3b025bc6c176fc3336fa"} Oct 30 22:09:20.361168 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:20.360659 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv" event={"ID":"312b05e0-c4aa-410f-b8ea-038f067ba880","Type":"ContainerStarted","Data":"5bbcef5489ec6eea4d12ce343a0f833e26c899df3155df9e5b9423138d8d4962"} Oct 30 22:09:20.361168 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:20.360689 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv" Oct 30 22:09:20.362316 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:20.362292 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52" event={"ID":"9138369d-c863-4773-b3d6-4060fa6f16f4","Type":"ContainerStarted","Data":"75c81f26d7a317bf278dd16dd186aaa508002faf5aa641cd02a3dfd635b2b18f"} Oct 30 22:09:20.362464 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:20.362343 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52" Oct 30 22:09:20.362464 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:20.362366 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52" event={"ID":"9138369d-c863-4773-b3d6-4060fa6f16f4","Type":"ContainerStarted","Data":"bc13eb90602a727dd4f1e59ab6e8a52ce52499f3e08bee0335666d3050b9d144"} Oct 30 22:09:20.379915 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:20.379619 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv" podStartSLOduration=1.379606554 podStartE2EDuration="1.379606554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:20.378598613 +0000 UTC m=+2428.974415298" watchObservedRunningTime="2026-10-30 22:09:20.379606554 +0000 UTC m=+2428.975423214" Oct 30 22:09:20.417011 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:20.415317 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52" podStartSLOduration=1.415301348 podStartE2EDuration="1.415301348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:20.411903032 +0000 UTC m=+2429.007719692" watchObservedRunningTime="2026-10-30 22:09:20.415301348 +0000 UTC m=+2429.011118008" Oct 30 22:09:20.818345 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:20.818309 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:21.676146 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:21.676113 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:22.649644 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:22.649607 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:23.615026 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:23.614983 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:24.324703 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:24.324667 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:25.202583 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:25.202545 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:25.950091 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:25.950056 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:26.178799 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:26.178737 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:26.809838 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:26.809797 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:27.611723 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:27.611686 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:28.607100 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:28.607068 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:29.343139 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:29.343102 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:30.094363 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:30.094323 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:30.278531 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:09:30.278488 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:09:31.000019 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:30.999976 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:31.371344 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:31.371302 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52" Oct 30 22:09:31.371837 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:31.371644 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv" Oct 30 22:09:31.775706 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:31.775623 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:32.345414 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:32.345385 5801 patch_prober.go:29] interesting pod/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 30 22:09:32.345582 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:32.345447 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" podUID="f781d69c-c4c6-4323-b8e3-28fccbf1b22f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 30 22:09:32.434906 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:32.434873 5801 generic.go:356] "Generic (PLEG): container finished" podID="f781d69c-c4c6-4323-b8e3-28fccbf1b22f" containerID="a55958e6adf52d2bec2098d46884922c5bfefef2751af60a1e4ffc3527cc1b52" exitCode=0 Oct 30 22:09:32.435308 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:32.434945 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" event={"ID":"f781d69c-c4c6-4323-b8e3-28fccbf1b22f","Type":"ContainerDied","Data":"a55958e6adf52d2bec2098d46884922c5bfefef2751af60a1e4ffc3527cc1b52"} Oct 30 22:09:32.435360 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:32.435315 5801 scope.go:122] "RemoveContainer" containerID="a55958e6adf52d2bec2098d46884922c5bfefef2751af60a1e4ffc3527cc1b52" Oct 30 22:09:32.436478 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:32.436452 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:09:32.436587 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:32.436535 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:09:32.436648 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:32.436582 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:09:32.739254 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:32.739144 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:33.442743 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:33.442706 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" event={"ID":"f781d69c-c4c6-4323-b8e3-28fccbf1b22f","Type":"ContainerStarted","Data":"894d6fb9b6c1003e9f20889790038524e056f7cff8fa425e4b7791071e3d41bd"} Oct 30 22:09:33.512524 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:33.512481 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:33.568257 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:33.568217 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:33.568655 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:33.568559 5801 patch_prober.go:29] interesting pod/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 30 22:09:33.568902 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:33.568878 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" podUID="f781d69c-c4c6-4323-b8e3-28fccbf1b22f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 30 22:09:33.967441 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:33.967396 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83f27219-hk78w"] Oct 30 22:09:33.982287 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:33.982259 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83f27219-hk78w"] Oct 30 22:09:33.982533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:33.982413 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83f27219-hk78w" Oct 30 22:09:33.986553 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:33.986527 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:09:33.986553 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:33.986547 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:09:33.986940 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:33.986921 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:09:33.988451 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:33.988423 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43805630-8c59-4f9b-8879-b317985a0dc0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-83f27219-hk78w\" (UID: \"43805630-8c59-4f9b-8879-b317985a0dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83f27219-hk78w" Oct 30 22:09:33.988624 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:33.988598 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjvt2\" (UniqueName: \"kubernetes.io/projected/43805630-8c59-4f9b-8879-b317985a0dc0-kube-api-access-wjvt2\") pod \"maas-api-revoke-keys-e2e-ait-83f27219-hk78w\" (UID: \"43805630-8c59-4f9b-8879-b317985a0dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83f27219-hk78w" Oct 30 22:09:33.988624 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:33.988539 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wg78n\"" Oct 30 22:09:33.988835 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:33.988687 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43805630-8c59-4f9b-8879-b317985a0dc0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-83f27219-hk78w\" (UID: \"43805630-8c59-4f9b-8879-b317985a0dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83f27219-hk78w" Oct 30 22:09:34.089468 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:34.089429 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43805630-8c59-4f9b-8879-b317985a0dc0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-83f27219-hk78w\" (UID: \"43805630-8c59-4f9b-8879-b317985a0dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83f27219-hk78w" Oct 30 22:09:34.089468 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:34.089471 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjvt2\" (UniqueName: \"kubernetes.io/projected/43805630-8c59-4f9b-8879-b317985a0dc0-kube-api-access-wjvt2\") pod \"maas-api-revoke-keys-e2e-ait-83f27219-hk78w\" (UID: \"43805630-8c59-4f9b-8879-b317985a0dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83f27219-hk78w" Oct 30 22:09:34.089724 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:34.089525 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43805630-8c59-4f9b-8879-b317985a0dc0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-83f27219-hk78w\" (UID: \"43805630-8c59-4f9b-8879-b317985a0dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83f27219-hk78w" Oct 30 22:09:34.090139 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:34.090115 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43805630-8c59-4f9b-8879-b317985a0dc0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-83f27219-hk78w\" (UID: \"43805630-8c59-4f9b-8879-b317985a0dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83f27219-hk78w" Oct 30 22:09:34.091896 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:34.091873 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43805630-8c59-4f9b-8879-b317985a0dc0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-83f27219-hk78w\" (UID: \"43805630-8c59-4f9b-8879-b317985a0dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83f27219-hk78w" Oct 30 22:09:34.096138 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:09:34.096120 5801 projected.go:196] Error preparing data for projected volume kube-api-access-wjvt2 for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83f27219-hk78w: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 30 22:09:34.096231 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:09:34.096203 5801 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43805630-8c59-4f9b-8879-b317985a0dc0-kube-api-access-wjvt2 podName:43805630-8c59-4f9b-8879-b317985a0dc0 nodeName:}" failed. No retries permitted until 2026-10-30 22:09:34.596188796 +0000 UTC m=+2443.192005433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wjvt2" (UniqueName: "kubernetes.io/projected/43805630-8c59-4f9b-8879-b317985a0dc0-kube-api-access-wjvt2") pod "maas-api-revoke-keys-e2e-ait-83f27219-hk78w" (UID: "43805630-8c59-4f9b-8879-b317985a0dc0") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 30 22:09:34.408205 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:34.408168 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:34.576776 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:34.573680 5801 patch_prober.go:29] interesting pod/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 30 22:09:34.576776 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:34.573740 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" podUID="f781d69c-c4c6-4323-b8e3-28fccbf1b22f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 30 22:09:34.697626 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:34.697523 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjvt2\" (UniqueName: \"kubernetes.io/projected/43805630-8c59-4f9b-8879-b317985a0dc0-kube-api-access-wjvt2\") pod \"maas-api-revoke-keys-e2e-ait-83f27219-hk78w\" (UID: \"43805630-8c59-4f9b-8879-b317985a0dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83f27219-hk78w" Oct 30 22:09:34.711269 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:34.711242 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjvt2\" (UniqueName: \"kubernetes.io/projected/43805630-8c59-4f9b-8879-b317985a0dc0-kube-api-access-wjvt2\") pod \"maas-api-revoke-keys-e2e-ait-83f27219-hk78w\" (UID: \"43805630-8c59-4f9b-8879-b317985a0dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83f27219-hk78w" Oct 30 22:09:34.897372 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:34.897330 5801 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83f27219-hk78w" secret="" err="secret \"maas-api-cleanup-dockercfg-wg78n\" not found" Oct 30 22:09:34.897540 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:34.897390 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83f27219-hk78w" Oct 30 22:09:35.193596 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:35.193568 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83f27219-hk78w"] Oct 30 22:09:35.195153 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:09:35.195113 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43805630_8c59_4f9b_8879_b317985a0dc0.slice/crio-58fd575ba56f37bf4d546be18d0cf68bdac08727fd9f7deb89b4d7e723a5e4b5 WatchSource:0}: Error finding container 58fd575ba56f37bf4d546be18d0cf68bdac08727fd9f7deb89b4d7e723a5e4b5: Status 404 returned error can't find the container with id 58fd575ba56f37bf4d546be18d0cf68bdac08727fd9f7deb89b4d7e723a5e4b5 Oct 30 22:09:35.327198 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:35.327161 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:35.481768 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:35.478173 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83f27219-hk78w" event={"ID":"43805630-8c59-4f9b-8879-b317985a0dc0","Type":"ContainerStarted","Data":"58fd575ba56f37bf4d546be18d0cf68bdac08727fd9f7deb89b4d7e723a5e4b5"} Oct 30 22:09:35.577777 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:35.575080 5801 patch_prober.go:29] interesting pod/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 30 22:09:35.577777 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:35.575132 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" podUID="f781d69c-c4c6-4323-b8e3-28fccbf1b22f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 30 22:09:36.127097 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:36.127063 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:36.486462 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:36.486379 5801 generic.go:356] "Generic (PLEG): container finished" podID="43805630-8c59-4f9b-8879-b317985a0dc0" containerID="678df26e8cd4177f73e6ff284f192df221f4c1ac22272ec5fe80f20ffabf8807" exitCode=0 Oct 30 22:09:36.486625 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:36.486462 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83f27219-hk78w" event={"ID":"43805630-8c59-4f9b-8879-b317985a0dc0","Type":"ContainerDied","Data":"678df26e8cd4177f73e6ff284f192df221f4c1ac22272ec5fe80f20ffabf8807"} Oct 30 22:09:36.575775 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:36.575078 5801 patch_prober.go:29] interesting pod/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 30 22:09:36.575775 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:36.575132 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" podUID="f781d69c-c4c6-4323-b8e3-28fccbf1b22f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 30 22:09:36.978717 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:36.978674 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:37.581817 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:37.573207 5801 patch_prober.go:29] interesting pod/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 30 22:09:37.581817 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:37.573272 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" podUID="f781d69c-c4c6-4323-b8e3-28fccbf1b22f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 30 22:09:37.636113 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:37.636087 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83f27219-hk78w" Oct 30 22:09:37.727134 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:37.727102 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43805630-8c59-4f9b-8879-b317985a0dc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43805630-8c59-4f9b-8879-b317985a0dc0-maas-api-xks-ca\") pod \"43805630-8c59-4f9b-8879-b317985a0dc0\" (UID: \"43805630-8c59-4f9b-8879-b317985a0dc0\") " Oct 30 22:09:37.727355 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:37.727178 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43805630-8c59-4f9b-8879-b317985a0dc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43805630-8c59-4f9b-8879-b317985a0dc0-maas-api-openshift-ca\") pod \"43805630-8c59-4f9b-8879-b317985a0dc0\" (UID: \"43805630-8c59-4f9b-8879-b317985a0dc0\") " Oct 30 22:09:37.727355 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:37.727320 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43805630-8c59-4f9b-8879-b317985a0dc0-kube-api-access-wjvt2\" (UniqueName: \"kubernetes.io/projected/43805630-8c59-4f9b-8879-b317985a0dc0-kube-api-access-wjvt2\") pod \"43805630-8c59-4f9b-8879-b317985a0dc0\" (UID: \"43805630-8c59-4f9b-8879-b317985a0dc0\") " Oct 30 22:09:37.727624 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:37.727600 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43805630-8c59-4f9b-8879-b317985a0dc0-maas-api-openshift-ca" pod "43805630-8c59-4f9b-8879-b317985a0dc0" (UID: "43805630-8c59-4f9b-8879-b317985a0dc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:37.729347 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:37.729316 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43805630-8c59-4f9b-8879-b317985a0dc0-maas-api-xks-ca" pod "43805630-8c59-4f9b-8879-b317985a0dc0" (UID: "43805630-8c59-4f9b-8879-b317985a0dc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:37.729462 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:37.729383 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43805630-8c59-4f9b-8879-b317985a0dc0-kube-api-access-wjvt2" pod "43805630-8c59-4f9b-8879-b317985a0dc0" (UID: "43805630-8c59-4f9b-8879-b317985a0dc0"). InnerVolumeSpecName "kube-api-access-wjvt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:37.787338 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:37.787234 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:37.828365 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:37.828329 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43805630-8c59-4f9b-8879-b317985a0dc0-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:37.828365 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:37.828361 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43805630-8c59-4f9b-8879-b317985a0dc0-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:37.828577 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:37.828376 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjvt2\" (UniqueName: \"kubernetes.io/projected/43805630-8c59-4f9b-8879-b317985a0dc0-kube-api-access-wjvt2\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:38.499537 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:38.499504 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83f27219-hk78w" Oct 30 22:09:38.499537 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:38.499533 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83f27219-hk78w" event={"ID":"43805630-8c59-4f9b-8879-b317985a0dc0","Type":"ContainerDied","Data":"58fd575ba56f37bf4d546be18d0cf68bdac08727fd9f7deb89b4d7e723a5e4b5"} Oct 30 22:09:38.500118 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:38.499602 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58fd575ba56f37bf4d546be18d0cf68bdac08727fd9f7deb89b4d7e723a5e4b5" Oct 30 22:09:38.568571 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:38.568541 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:38.568941 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:38.568919 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:38.569707 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:38.569690 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:38.577666 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:38.577640 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:39.433583 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:39.433514 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:40.092040 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:40.092007 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:40.987320 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:40.987264 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:41.676456 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:41.676423 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:42.382714 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:42.382670 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:43.177516 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:43.177485 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:43.888433 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:43.888396 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:44.084590 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:44.084554 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52"] Oct 30 22:09:44.103481 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:44.103450 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv"] Oct 30 22:09:44.225217 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:44.225140 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:44.347117 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:44.347081 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:44.584650 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:44.584613 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:44.792359 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:44.792324 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:44.896676 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:44.896602 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83f27219-hk78w"] Oct 30 22:09:44.901376 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:44.901344 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83f27219-hk78w"] Oct 30 22:09:44.949321 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:44.949284 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:45.522533 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:45.522493 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43805630-8c59-4f9b-8879-b317985a0dc0" path="/var/lib/kubelet/pods/43805630-8c59-4f9b-8879-b317985a0dc0/volumes" Oct 30 22:09:49.085063 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.085015 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52" podUID="9138369d-c863-4773-b3d6-4060fa6f16f4" containerName="payload-processing" containerID="cri-o://75c81f26d7a317bf278dd16dd186aaa508002faf5aa641cd02a3dfd635b2b18f" gracePeriod=25 Oct 30 22:09:49.104662 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.104633 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv" podUID="312b05e0-c4aa-410f-b8ea-038f067ba880" containerName="payload-pre-processing" containerID="cri-o://5ecf9c084c9414fa9c8b53f18dc2b539dff6866ae5fd3b025bc6c176fc3336fa" gracePeriod=25 Oct 30 22:09:49.356038 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.355981 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52" Oct 30 22:09:49.375561 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.375539 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv" Oct 30 22:09:49.433824 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.433796 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9138369d-c863-4773-b3d6-4060fa6f16f4-kube-api-access-z2bzq\" (UniqueName: \"kubernetes.io/projected/9138369d-c863-4773-b3d6-4060fa6f16f4-kube-api-access-z2bzq\") pod \"9138369d-c863-4773-b3d6-4060fa6f16f4\" (UID: \"9138369d-c863-4773-b3d6-4060fa6f16f4\") " Oct 30 22:09:49.433966 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.433879 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/312b05e0-c4aa-410f-b8ea-038f067ba880-kube-api-access-tmff2\" (UniqueName: \"kubernetes.io/projected/312b05e0-c4aa-410f-b8ea-038f067ba880-kube-api-access-tmff2\") pod \"312b05e0-c4aa-410f-b8ea-038f067ba880\" (UID: \"312b05e0-c4aa-410f-b8ea-038f067ba880\") " Oct 30 22:09:49.433966 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.433912 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9138369d-c863-4773-b3d6-4060fa6f16f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9138369d-c863-4773-b3d6-4060fa6f16f4-plugins-config-volume\") pod \"9138369d-c863-4773-b3d6-4060fa6f16f4\" (UID: \"9138369d-c863-4773-b3d6-4060fa6f16f4\") " Oct 30 22:09:49.434104 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.433969 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/312b05e0-c4aa-410f-b8ea-038f067ba880-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312b05e0-c4aa-410f-b8ea-038f067ba880-plugins-config-volume\") pod \"312b05e0-c4aa-410f-b8ea-038f067ba880\" (UID: \"312b05e0-c4aa-410f-b8ea-038f067ba880\") " Oct 30 22:09:49.434343 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.434317 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9138369d-c863-4773-b3d6-4060fa6f16f4-plugins-config-volume" pod "9138369d-c863-4773-b3d6-4060fa6f16f4" (UID: "9138369d-c863-4773-b3d6-4060fa6f16f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:49.434426 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.434369 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/312b05e0-c4aa-410f-b8ea-038f067ba880-plugins-config-volume" pod "312b05e0-c4aa-410f-b8ea-038f067ba880" (UID: "312b05e0-c4aa-410f-b8ea-038f067ba880"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:49.435715 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.435687 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9138369d-c863-4773-b3d6-4060fa6f16f4-kube-api-access-z2bzq" pod "9138369d-c863-4773-b3d6-4060fa6f16f4" (UID: "9138369d-c863-4773-b3d6-4060fa6f16f4"). InnerVolumeSpecName "kube-api-access-z2bzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:49.435854 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.435834 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/312b05e0-c4aa-410f-b8ea-038f067ba880-kube-api-access-tmff2" pod "312b05e0-c4aa-410f-b8ea-038f067ba880" (UID: "312b05e0-c4aa-410f-b8ea-038f067ba880"). InnerVolumeSpecName "kube-api-access-tmff2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:49.534855 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.534834 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2bzq\" (UniqueName: \"kubernetes.io/projected/9138369d-c863-4773-b3d6-4060fa6f16f4-kube-api-access-z2bzq\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:49.534855 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.534856 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmff2\" (UniqueName: \"kubernetes.io/projected/312b05e0-c4aa-410f-b8ea-038f067ba880-kube-api-access-tmff2\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:49.534986 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.534866 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9138369d-c863-4773-b3d6-4060fa6f16f4-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:49.534986 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.534875 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312b05e0-c4aa-410f-b8ea-038f067ba880-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:49.560171 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.560143 5801 generic.go:356] "Generic (PLEG): container finished" podID="312b05e0-c4aa-410f-b8ea-038f067ba880" containerID="5ecf9c084c9414fa9c8b53f18dc2b539dff6866ae5fd3b025bc6c176fc3336fa" exitCode=0 Oct 30 22:09:49.560280 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.560182 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv" Oct 30 22:09:49.560280 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.560228 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv" event={"ID":"312b05e0-c4aa-410f-b8ea-038f067ba880","Type":"ContainerDied","Data":"5ecf9c084c9414fa9c8b53f18dc2b539dff6866ae5fd3b025bc6c176fc3336fa"} Oct 30 22:09:49.560280 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.560270 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv" event={"ID":"312b05e0-c4aa-410f-b8ea-038f067ba880","Type":"ContainerDied","Data":"5bbcef5489ec6eea4d12ce343a0f833e26c899df3155df9e5b9423138d8d4962"} Oct 30 22:09:49.560447 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.560293 5801 scope.go:122] "RemoveContainer" containerID="5ecf9c084c9414fa9c8b53f18dc2b539dff6866ae5fd3b025bc6c176fc3336fa" Oct 30 22:09:49.561576 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.561554 5801 generic.go:356] "Generic (PLEG): container finished" podID="9138369d-c863-4773-b3d6-4060fa6f16f4" containerID="75c81f26d7a317bf278dd16dd186aaa508002faf5aa641cd02a3dfd635b2b18f" exitCode=0 Oct 30 22:09:49.561671 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.561595 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52" Oct 30 22:09:49.561739 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.561641 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52" event={"ID":"9138369d-c863-4773-b3d6-4060fa6f16f4","Type":"ContainerDied","Data":"75c81f26d7a317bf278dd16dd186aaa508002faf5aa641cd02a3dfd635b2b18f"} Oct 30 22:09:49.561739 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.561713 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52" event={"ID":"9138369d-c863-4773-b3d6-4060fa6f16f4","Type":"ContainerDied","Data":"bc13eb90602a727dd4f1e59ab6e8a52ce52499f3e08bee0335666d3050b9d144"} Oct 30 22:09:49.569380 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.569365 5801 scope.go:122] "RemoveContainer" containerID="5ecf9c084c9414fa9c8b53f18dc2b539dff6866ae5fd3b025bc6c176fc3336fa" Oct 30 22:09:49.569668 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:09:49.569644 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ecf9c084c9414fa9c8b53f18dc2b539dff6866ae5fd3b025bc6c176fc3336fa\": container with ID starting with 5ecf9c084c9414fa9c8b53f18dc2b539dff6866ae5fd3b025bc6c176fc3336fa not found: ID does not exist" containerID="5ecf9c084c9414fa9c8b53f18dc2b539dff6866ae5fd3b025bc6c176fc3336fa" Oct 30 22:09:49.569810 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.569672 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ecf9c084c9414fa9c8b53f18dc2b539dff6866ae5fd3b025bc6c176fc3336fa"} err="failed to get container status \"5ecf9c084c9414fa9c8b53f18dc2b539dff6866ae5fd3b025bc6c176fc3336fa\": rpc error: code = NotFound desc = could not find container \"5ecf9c084c9414fa9c8b53f18dc2b539dff6866ae5fd3b025bc6c176fc3336fa\": container with ID starting with 5ecf9c084c9414fa9c8b53f18dc2b539dff6866ae5fd3b025bc6c176fc3336fa not found: ID does not exist" Oct 30 22:09:49.569810 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.569689 5801 scope.go:122] "RemoveContainer" containerID="75c81f26d7a317bf278dd16dd186aaa508002faf5aa641cd02a3dfd635b2b18f" Oct 30 22:09:49.577059 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.577038 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv"] Oct 30 22:09:49.578031 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.578012 5801 scope.go:122] "RemoveContainer" containerID="75c81f26d7a317bf278dd16dd186aaa508002faf5aa641cd02a3dfd635b2b18f" Oct 30 22:09:49.578883 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:09:49.578856 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c81f26d7a317bf278dd16dd186aaa508002faf5aa641cd02a3dfd635b2b18f\": container with ID starting with 75c81f26d7a317bf278dd16dd186aaa508002faf5aa641cd02a3dfd635b2b18f not found: ID does not exist" containerID="75c81f26d7a317bf278dd16dd186aaa508002faf5aa641cd02a3dfd635b2b18f" Oct 30 22:09:49.578984 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.578890 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c81f26d7a317bf278dd16dd186aaa508002faf5aa641cd02a3dfd635b2b18f"} err="failed to get container status \"75c81f26d7a317bf278dd16dd186aaa508002faf5aa641cd02a3dfd635b2b18f\": rpc error: code = NotFound desc = could not find container \"75c81f26d7a317bf278dd16dd186aaa508002faf5aa641cd02a3dfd635b2b18f\": container with ID starting with 75c81f26d7a317bf278dd16dd186aaa508002faf5aa641cd02a3dfd635b2b18f not found: ID does not exist" Oct 30 22:09:49.580727 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.580710 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-83f27219-7f58cc86ff-7vcgv"] Oct 30 22:09:49.593349 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.593324 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52"] Oct 30 22:09:49.595152 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:49.595133 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-83f27219-65cff5b6cf-xnn52"] Oct 30 22:09:50.416221 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:50.416139 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt"] Oct 30 22:09:50.416671 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:50.416405 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" podUID="f781d69c-c4c6-4323-b8e3-28fccbf1b22f" containerName="istio-proxy" containerID="cri-o://894d6fb9b6c1003e9f20889790038524e056f7cff8fa425e4b7791071e3d41bd" gracePeriod=30 Oct 30 22:09:50.468075 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:50.468044 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:51.522114 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:51.522078 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="312b05e0-c4aa-410f-b8ea-038f067ba880" path="/var/lib/kubelet/pods/312b05e0-c4aa-410f-b8ea-038f067ba880/volumes" Oct 30 22:09:51.522642 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:51.522621 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9138369d-c863-4773-b3d6-4060fa6f16f4" path="/var/lib/kubelet/pods/9138369d-c863-4773-b3d6-4060fa6f16f4/volumes" Oct 30 22:09:51.831102 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:51.831069 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:52.243947 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.243865 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl"] Oct 30 22:09:52.244320 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.244300 5801 cpu_manager.go:414] "containerMap: removing container" podUID="312b05e0-c4aa-410f-b8ea-038f067ba880" containerName="payload-pre-processing" Oct 30 22:09:52.244415 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.244322 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="312b05e0-c4aa-410f-b8ea-038f067ba880" containerName="payload-pre-processing" Oct 30 22:09:52.244415 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.244349 5801 cpu_manager.go:414] "containerMap: removing container" podUID="9138369d-c863-4773-b3d6-4060fa6f16f4" containerName="payload-processing" Oct 30 22:09:52.244415 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.244357 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9138369d-c863-4773-b3d6-4060fa6f16f4" containerName="payload-processing" Oct 30 22:09:52.244415 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.244387 5801 cpu_manager.go:414] "containerMap: removing container" podUID="43805630-8c59-4f9b-8879-b317985a0dc0" containerName="revoke-keys" Oct 30 22:09:52.244415 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.244396 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43805630-8c59-4f9b-8879-b317985a0dc0" containerName="revoke-keys" Oct 30 22:09:52.244687 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.244505 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="312b05e0-c4aa-410f-b8ea-038f067ba880" containerName="payload-pre-processing" Oct 30 22:09:52.244687 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.244521 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="9138369d-c863-4773-b3d6-4060fa6f16f4" containerName="payload-processing" Oct 30 22:09:52.244687 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.244535 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="43805630-8c59-4f9b-8879-b317985a0dc0" containerName="revoke-keys" Oct 30 22:09:52.249097 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.249074 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.256729 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.256702 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-7766bb-openshift-default-dockercfg-7pjv6\"" Oct 30 22:09:52.271678 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.271211 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl"] Oct 30 22:09:52.307685 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.307653 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:52.359931 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.359902 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-token\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.360080 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.359937 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-envoy\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.360080 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.359956 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-workload-socket\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.360080 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.360040 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-podinfo\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.360273 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.360117 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-credential-socket\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.360273 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.360223 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-data\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.360273 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.360268 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-workload-certs\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.360400 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.360310 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltk22\" (UniqueName: \"kubernetes.io/projected/e81f465f-d0b9-48c0-a7cc-21acb683c890-kube-api-access-ltk22\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.360400 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.360331 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e81f465f-d0b9-48c0-a7cc-21acb683c890-istiod-ca-cert\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.413627 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.413587 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:52.461695 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.461665 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-token\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.461886 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.461703 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-envoy\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.461886 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.461721 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-workload-socket\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.461886 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.461743 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-podinfo\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.461886 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.461802 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-credential-socket\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.461886 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.461882 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-data\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.462206 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.461921 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-workload-certs\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.462206 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.461948 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltk22\" (UniqueName: \"kubernetes.io/projected/e81f465f-d0b9-48c0-a7cc-21acb683c890-kube-api-access-ltk22\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.462206 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.461977 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e81f465f-d0b9-48c0-a7cc-21acb683c890-istiod-ca-cert\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.462394 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.462205 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-workload-socket\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.462394 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.462267 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-credential-socket\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.462580 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.462556 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-data\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.462662 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.462600 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-workload-certs\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.462828 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.462805 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e81f465f-d0b9-48c0-a7cc-21acb683c890-istiod-ca-cert\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.464147 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.464131 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-envoy\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.464475 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.464452 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-podinfo\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.474573 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.474543 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-token\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.474674 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.474633 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltk22\" (UniqueName: \"kubernetes.io/projected/e81f465f-d0b9-48c0-a7cc-21acb683c890-kube-api-access-ltk22\") pod \"e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.562745 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.562712 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:52.844483 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.844403 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl"] Oct 30 22:09:52.849497 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.849464 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:09:52.849578 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.849548 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:09:52.849621 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:52.849576 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:09:53.571500 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:53.571465 5801 patch_prober.go:29] interesting pod/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:09:53.571905 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:53.571526 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" podUID="f781d69c-c4c6-4323-b8e3-28fccbf1b22f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:09:53.586229 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:53.586197 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" event={"ID":"e81f465f-d0b9-48c0-a7cc-21acb683c890","Type":"ContainerStarted","Data":"c2dd91de765143b7bbce2df6fa759330d7409f960d69fda6023b8c0848dcd2fe"} Oct 30 22:09:53.586373 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:53.586258 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" event={"ID":"e81f465f-d0b9-48c0-a7cc-21acb683c890","Type":"ContainerStarted","Data":"e3455978a47d9abf13101702af7ffd1510bbd94787b408bdc4398d7d81c55fde"} Oct 30 22:09:53.613202 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:53.613150 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" podStartSLOduration=1.613137975 podStartE2EDuration="1.613137975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:53.610729652 +0000 UTC m=+2462.206546313" watchObservedRunningTime="2026-10-30 22:09:53.613137975 +0000 UTC m=+2462.208954634" Oct 30 22:09:54.563217 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:54.563184 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:54.568253 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:54.568226 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:54.592341 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:54.592310 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:54.593546 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:54.593526 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:09:55.520705 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:09:55.520674 5801 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf781d69c_c4c6_4323_b8e3_28fccbf1b22f.slice/crio-72356c848f262dcbe357f3153bd0ab18ed7d3ee0262e5265bf62689c31c3e69d\": RecentStats: unable to find data in memory cache]" Oct 30 22:09:55.601632 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.601607 5801 generic.go:356] "Generic (PLEG): container finished" podID="f781d69c-c4c6-4323-b8e3-28fccbf1b22f" containerID="894d6fb9b6c1003e9f20889790038524e056f7cff8fa425e4b7791071e3d41bd" exitCode=0 Oct 30 22:09:55.602017 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.601682 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" event={"ID":"f781d69c-c4c6-4323-b8e3-28fccbf1b22f","Type":"ContainerDied","Data":"894d6fb9b6c1003e9f20889790038524e056f7cff8fa425e4b7791071e3d41bd"} Oct 30 22:09:55.602017 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.601745 5801 scope.go:122] "RemoveContainer" containerID="a55958e6adf52d2bec2098d46884922c5bfefef2751af60a1e4ffc3527cc1b52" Oct 30 22:09:55.667608 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.667588 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:55.793281 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.793125 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-data\") pod \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " Oct 30 22:09:55.793281 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.793182 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-podinfo\") pod \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " Oct 30 22:09:55.793281 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.793256 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-workload-socket\") pod \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " Oct 30 22:09:55.793539 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.793319 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-kube-api-access-hngxw\" (UniqueName: \"kubernetes.io/projected/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-kube-api-access-hngxw\") pod \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " Oct 30 22:09:55.793539 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.793362 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-envoy\") pod \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " Oct 30 22:09:55.793539 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.793398 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-workload-certs\") pod \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " Oct 30 22:09:55.793539 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.793457 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-token\" (UniqueName: \"kubernetes.io/projected/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-token\") pod \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " Oct 30 22:09:55.793539 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.793494 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-credential-socket\") pod \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " Oct 30 22:09:55.793539 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.793525 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istiod-ca-cert\") pod \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\" (UID: \"f781d69c-c4c6-4323-b8e3-28fccbf1b22f\") " Oct 30 22:09:55.793539 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.793524 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-data" pod "f781d69c-c4c6-4323-b8e3-28fccbf1b22f" (UID: "f781d69c-c4c6-4323-b8e3-28fccbf1b22f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:55.793860 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.793840 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:55.793913 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.793857 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-workload-certs" pod "f781d69c-c4c6-4323-b8e3-28fccbf1b22f" (UID: "f781d69c-c4c6-4323-b8e3-28fccbf1b22f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:55.794309 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.794154 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istiod-ca-cert" pod "f781d69c-c4c6-4323-b8e3-28fccbf1b22f" (UID: "f781d69c-c4c6-4323-b8e3-28fccbf1b22f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:55.794309 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.794156 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-workload-socket" pod "f781d69c-c4c6-4323-b8e3-28fccbf1b22f" (UID: "f781d69c-c4c6-4323-b8e3-28fccbf1b22f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:55.794474 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.794316 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-credential-socket" pod "f781d69c-c4c6-4323-b8e3-28fccbf1b22f" (UID: "f781d69c-c4c6-4323-b8e3-28fccbf1b22f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:55.795939 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.795916 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-kube-api-access-hngxw" pod "f781d69c-c4c6-4323-b8e3-28fccbf1b22f" (UID: "f781d69c-c4c6-4323-b8e3-28fccbf1b22f"). InnerVolumeSpecName "kube-api-access-hngxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:55.796189 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.795978 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-token" pod "f781d69c-c4c6-4323-b8e3-28fccbf1b22f" (UID: "f781d69c-c4c6-4323-b8e3-28fccbf1b22f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:55.796189 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.796136 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-envoy" pod "f781d69c-c4c6-4323-b8e3-28fccbf1b22f" (UID: "f781d69c-c4c6-4323-b8e3-28fccbf1b22f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:55.796189 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.796142 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-podinfo" pod "f781d69c-c4c6-4323-b8e3-28fccbf1b22f" (UID: "f781d69c-c4c6-4323-b8e3-28fccbf1b22f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:09:55.893012 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.892983 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:55.895100 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.895077 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:55.895100 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.895099 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:55.895286 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.895110 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hngxw\" (UniqueName: \"kubernetes.io/projected/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-kube-api-access-hngxw\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:55.895286 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.895121 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:55.895286 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.895132 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:55.895286 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.895141 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:55.895286 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.895149 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:55.895286 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:55.895160 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f781d69c-c4c6-4323-b8e3-28fccbf1b22f-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:56.015212 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:56.015181 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:56.611638 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:56.611608 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" Oct 30 22:09:56.612139 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:56.611635 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt" event={"ID":"f781d69c-c4c6-4323-b8e3-28fccbf1b22f","Type":"ContainerDied","Data":"72356c848f262dcbe357f3153bd0ab18ed7d3ee0262e5265bf62689c31c3e69d"} Oct 30 22:09:56.612139 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:56.611715 5801 scope.go:122] "RemoveContainer" containerID="894d6fb9b6c1003e9f20889790038524e056f7cff8fa425e4b7791071e3d41bd" Oct 30 22:09:56.637685 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:56.637658 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt"] Oct 30 22:09:56.642439 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:56.642414 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-83f27219-openshift-default-784db9f5f7-cd8xt"] Oct 30 22:09:57.080532 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.080497 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:57.219512 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.219476 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:57.228711 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.228682 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww"] Oct 30 22:09:57.229097 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.229084 5801 cpu_manager.go:414] "containerMap: removing container" podUID="f781d69c-c4c6-4323-b8e3-28fccbf1b22f" containerName="istio-proxy" Oct 30 22:09:57.229147 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.229099 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f781d69c-c4c6-4323-b8e3-28fccbf1b22f" containerName="istio-proxy" Oct 30 22:09:57.229195 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.229183 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="f781d69c-c4c6-4323-b8e3-28fccbf1b22f" containerName="istio-proxy" Oct 30 22:09:57.233710 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.233695 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww" Oct 30 22:09:57.240358 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.240321 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-7766bb\"" Oct 30 22:09:57.242790 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.242765 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-7766bb-dockercfg-qfkfp\"" Oct 30 22:09:57.243279 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.243259 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww"] Oct 30 22:09:57.307492 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.307452 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb5fbb2-783c-4921-a2ad-8edf9bf67a27-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww\" (UID: \"9eb5fbb2-783c-4921-a2ad-8edf9bf67a27\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww" Oct 30 22:09:57.307878 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.307848 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67vjj\" (UniqueName: \"kubernetes.io/projected/9eb5fbb2-783c-4921-a2ad-8edf9bf67a27-kube-api-access-67vjj\") pod \"payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww\" (UID: \"9eb5fbb2-783c-4921-a2ad-8edf9bf67a27\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww" Oct 30 22:09:57.325658 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.325620 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:57.386518 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.386443 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j"] Oct 30 22:09:57.386873 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.386859 5801 cpu_manager.go:414] "containerMap: removing container" podUID="f781d69c-c4c6-4323-b8e3-28fccbf1b22f" containerName="istio-proxy" Oct 30 22:09:57.386941 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.386874 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f781d69c-c4c6-4323-b8e3-28fccbf1b22f" containerName="istio-proxy" Oct 30 22:09:57.386982 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.386942 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="f781d69c-c4c6-4323-b8e3-28fccbf1b22f" containerName="istio-proxy" Oct 30 22:09:57.390019 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.390005 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j" Oct 30 22:09:57.408626 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.408589 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb5fbb2-783c-4921-a2ad-8edf9bf67a27-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww\" (UID: \"9eb5fbb2-783c-4921-a2ad-8edf9bf67a27\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww" Oct 30 22:09:57.408776 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.408651 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67vjj\" (UniqueName: \"kubernetes.io/projected/9eb5fbb2-783c-4921-a2ad-8edf9bf67a27-kube-api-access-67vjj\") pod \"payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww\" (UID: \"9eb5fbb2-783c-4921-a2ad-8edf9bf67a27\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww" Oct 30 22:09:57.409230 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.409211 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb5fbb2-783c-4921-a2ad-8edf9bf67a27-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww\" (UID: \"9eb5fbb2-783c-4921-a2ad-8edf9bf67a27\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww" Oct 30 22:09:57.425248 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.425224 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j"] Oct 30 22:09:57.438990 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.438968 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67vjj\" (UniqueName: \"kubernetes.io/projected/9eb5fbb2-783c-4921-a2ad-8edf9bf67a27-kube-api-access-67vjj\") pod \"payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww\" (UID: \"9eb5fbb2-783c-4921-a2ad-8edf9bf67a27\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww" Oct 30 22:09:57.509706 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.509666 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5f9638-8e4e-4199-ac2c-23802b71407a-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j\" (UID: \"9e5f9638-8e4e-4199-ac2c-23802b71407a\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j" Oct 30 22:09:57.509889 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.509729 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbpqx\" (UniqueName: \"kubernetes.io/projected/9e5f9638-8e4e-4199-ac2c-23802b71407a-kube-api-access-zbpqx\") pod \"payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j\" (UID: \"9e5f9638-8e4e-4199-ac2c-23802b71407a\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j" Oct 30 22:09:57.522154 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.522132 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f781d69c-c4c6-4323-b8e3-28fccbf1b22f" path="/var/lib/kubelet/pods/f781d69c-c4c6-4323-b8e3-28fccbf1b22f/volumes" Oct 30 22:09:57.549226 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.549204 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww" Oct 30 22:09:57.613057 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.612718 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbpqx\" (UniqueName: \"kubernetes.io/projected/9e5f9638-8e4e-4199-ac2c-23802b71407a-kube-api-access-zbpqx\") pod \"payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j\" (UID: \"9e5f9638-8e4e-4199-ac2c-23802b71407a\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j" Oct 30 22:09:57.613057 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.612950 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5f9638-8e4e-4199-ac2c-23802b71407a-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j\" (UID: \"9e5f9638-8e4e-4199-ac2c-23802b71407a\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j" Oct 30 22:09:57.613861 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.613834 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5f9638-8e4e-4199-ac2c-23802b71407a-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j\" (UID: \"9e5f9638-8e4e-4199-ac2c-23802b71407a\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j" Oct 30 22:09:57.626383 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.626359 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbpqx\" (UniqueName: \"kubernetes.io/projected/9e5f9638-8e4e-4199-ac2c-23802b71407a-kube-api-access-zbpqx\") pod \"payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j\" (UID: \"9e5f9638-8e4e-4199-ac2c-23802b71407a\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j" Oct 30 22:09:57.702371 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.702314 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j" Oct 30 22:09:57.827468 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.827441 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww"] Oct 30 22:09:57.828351 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:09:57.828330 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb5fbb2_783c_4921_a2ad_8edf9bf67a27.slice/crio-923f558f86e2d37965936b0812303ce4c97440c99acbea634d2e3d40b91db6e6 WatchSource:0}: Error finding container 923f558f86e2d37965936b0812303ce4c97440c99acbea634d2e3d40b91db6e6: Status 404 returned error can't find the container with id 923f558f86e2d37965936b0812303ce4c97440c99acbea634d2e3d40b91db6e6 Oct 30 22:09:57.982415 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:57.982393 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j"] Oct 30 22:09:57.984375 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:09:57.984352 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5f9638_8e4e_4199_ac2c_23802b71407a.slice/crio-714eef8ffc6bb350b9c333fdf964341f572cfc1e9dba6145d1543f54cd9736b5 WatchSource:0}: Error finding container 714eef8ffc6bb350b9c333fdf964341f572cfc1e9dba6145d1543f54cd9736b5: Status 404 returned error can't find the container with id 714eef8ffc6bb350b9c333fdf964341f572cfc1e9dba6145d1543f54cd9736b5 Oct 30 22:09:58.177221 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:58.177189 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:58.628181 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:58.628150 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww" event={"ID":"9eb5fbb2-783c-4921-a2ad-8edf9bf67a27","Type":"ContainerStarted","Data":"1dda43baf930733f09a9c334e5930f0758b825ccb2cd9d605b2ec1a6308040d1"} Oct 30 22:09:58.628647 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:58.628203 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww" Oct 30 22:09:58.628647 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:58.628225 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww" event={"ID":"9eb5fbb2-783c-4921-a2ad-8edf9bf67a27","Type":"ContainerStarted","Data":"923f558f86e2d37965936b0812303ce4c97440c99acbea634d2e3d40b91db6e6"} Oct 30 22:09:58.629635 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:58.629611 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j" event={"ID":"9e5f9638-8e4e-4199-ac2c-23802b71407a","Type":"ContainerStarted","Data":"2763df86e40ba0d17c7be5c16f84bede2c63c7fd3050721ef070c7ef36674ef7"} Oct 30 22:09:58.629731 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:58.629652 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j" event={"ID":"9e5f9638-8e4e-4199-ac2c-23802b71407a","Type":"ContainerStarted","Data":"714eef8ffc6bb350b9c333fdf964341f572cfc1e9dba6145d1543f54cd9736b5"} Oct 30 22:09:58.629731 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:58.629677 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j" Oct 30 22:09:58.649272 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:58.649232 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww" podStartSLOduration=1.6492197069999999 podStartE2EDuration="1.649219707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:58.646528677 +0000 UTC m=+2467.242345334" watchObservedRunningTime="2026-10-30 22:09:58.649219707 +0000 UTC m=+2467.245036366" Oct 30 22:09:58.668046 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:58.668010 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j" podStartSLOduration=1.6679986100000002 podStartE2EDuration="1.66799861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:58.666335602 +0000 UTC m=+2467.262152273" watchObservedRunningTime="2026-10-30 22:09:58.66799861 +0000 UTC m=+2467.263815269" Oct 30 22:09:58.889392 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:58.889297 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:59.212963 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:59.212884 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:59.458302 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:59.458272 5801 scope.go:122] "RemoveContainer" containerID="0df3de87c3265a833554562c399086ac584a16ffa87d3df4f14234ce89991261" Oct 30 22:09:59.467391 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:59.467370 5801 scope.go:122] "RemoveContainer" containerID="2bbdc1fbafe5dab3d1da23debbb43a86958ea89dbf641c970a2606ecb9b47df6" Oct 30 22:09:59.788709 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:09:59.788618 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:00.279070 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:10:00.279032 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:10:00.693273 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:00.693237 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:01.389184 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:01.389132 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:02.103534 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:02.103481 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:02.895190 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:02.895152 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:03.569234 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:03.569200 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:04.285978 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:04.285932 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:04.989859 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:04.989828 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:05.844496 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:05.844458 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:06.679658 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:06.679618 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:07.600788 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:07.600730 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:08.362104 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:08.362069 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:09.002359 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:09.002322 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:09.593989 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:09.593954 5801 patch_prober.go:29] interesting pod/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body= Oct 30 22:10:09.594168 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:09.594002 5801 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" podUID="e81f465f-d0b9-48c0-a7cc-21acb683c890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" Oct 30 22:10:09.637195 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:09.637167 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww" Oct 30 22:10:09.637389 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:09.637376 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j" Oct 30 22:10:09.711965 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:09.711930 5801 generic.go:356] "Generic (PLEG): container finished" podID="e81f465f-d0b9-48c0-a7cc-21acb683c890" containerID="c2dd91de765143b7bbce2df6fa759330d7409f960d69fda6023b8c0848dcd2fe" exitCode=0 Oct 30 22:10:09.711965 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:09.711979 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" event={"ID":"e81f465f-d0b9-48c0-a7cc-21acb683c890","Type":"ContainerDied","Data":"c2dd91de765143b7bbce2df6fa759330d7409f960d69fda6023b8c0848dcd2fe"} Oct 30 22:10:09.712359 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:09.712343 5801 scope.go:122] "RemoveContainer" containerID="c2dd91de765143b7bbce2df6fa759330d7409f960d69fda6023b8c0848dcd2fe" Oct 30 22:10:09.714313 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:09.714270 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:10:09.714421 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:09.714355 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:10:09.714421 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:09.714391 5801 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:10:09.831721 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:09.831685 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:10.580946 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:10.580904 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:10.719447 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:10.719416 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" event={"ID":"e81f465f-d0b9-48c0-a7cc-21acb683c890","Type":"ContainerStarted","Data":"7b2fc290dc2edb0451dd80619eba48c1c5b8555c0f79d623e36f088ba2e95257"} Oct 30 22:10:10.917784 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:10.917636 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:11.481273 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:11.481146 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:11.569206 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:11.569170 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:10:11.569560 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:11.569535 5801 patch_prober.go:29] interesting pod/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body= Oct 30 22:10:11.569721 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:11.569696 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" podUID="e81f465f-d0b9-48c0-a7cc-21acb683c890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" Oct 30 22:10:12.287329 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:12.287295 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:12.564119 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:12.564082 5801 patch_prober.go:29] interesting pod/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body= Oct 30 22:10:12.564940 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:12.564909 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" podUID="e81f465f-d0b9-48c0-a7cc-21acb683c890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" Oct 30 22:10:13.193972 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:13.193937 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:13.563113 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:13.563079 5801 patch_prober.go:29] interesting pod/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body= Oct 30 22:10:13.563685 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:13.563657 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" podUID="e81f465f-d0b9-48c0-a7cc-21acb683c890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" Oct 30 22:10:13.979394 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:13.979302 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:14.573791 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:14.573096 5801 patch_prober.go:29] interesting pod/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body= Oct 30 22:10:14.573791 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:14.573159 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" podUID="e81f465f-d0b9-48c0-a7cc-21acb683c890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" Oct 30 22:10:14.682482 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:14.682444 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:15.563729 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:15.563692 5801 patch_prober.go:29] interesting pod/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body= Oct 30 22:10:15.563993 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:15.563775 5801 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" podUID="e81f465f-d0b9-48c0-a7cc-21acb683c890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" Oct 30 22:10:15.603913 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:15.603871 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:16.385106 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:16.385065 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:16.566905 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:16.566875 5801 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:10:16.567169 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:16.567149 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:10:16.567806 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:16.567790 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:10:17.190531 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:17.190494 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:18.200801 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:18.200736 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:18.893967 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:18.893927 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:19.706769 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:19.706712 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:20.487984 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:20.487943 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:21.196784 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:21.196730 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:21.958927 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:21.958885 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:22.707930 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:22.707894 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:23.534789 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:23.534729 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:24.297171 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:24.297128 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:24.984582 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:24.984538 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:25.671057 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:25.671022 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:26.368298 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:26.368254 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:27.090960 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:27.090921 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:27.791822 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:27.791781 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:28.494582 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:28.494545 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:29.184227 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:29.184193 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:29.893737 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:29.893689 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:30.278182 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:10:30.278100 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:10:30.684604 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:30.684569 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:31.388221 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:31.388171 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:32.086578 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:32.086541 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:32.790174 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:32.790139 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:33.641239 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:33.641198 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:34.364944 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:34.364902 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:34.989316 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:34.989275 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:35.677437 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:35.677406 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:36.382725 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:36.382688 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:37.195220 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:37.195179 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:37.879550 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:37.879516 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:38.584645 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:38.584607 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:39.282103 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:39.282070 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:39.985140 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:39.985104 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:40.802498 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:40.800127 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:41.491897 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:41.491846 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:42.169063 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:42.169027 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:42.872775 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:42.872710 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:43.593921 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:43.593877 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:44.277563 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:44.277527 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:44.979208 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:44.979169 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:45.692506 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:45.692469 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:46.377350 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:46.377309 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:47.087932 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:47.087903 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:47.879003 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:47.878965 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:48.580144 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:48.580103 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:49.284416 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:49.284386 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:50.053783 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:50.053722 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:50.789691 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:50.789662 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:51.034602 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:51.034566 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d5rc9"] Oct 30 22:10:51.036631 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:51.036609 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-d5rc9"] Oct 30 22:10:51.492384 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:51.492343 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:51.523043 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:51.523006 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="620fbb96-a894-4901-89fc-55a0950a0c8d" path="/var/lib/kubelet/pods/620fbb96-a894-4901-89fc-55a0950a0c8d/volumes" Oct 30 22:10:52.273535 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:52.273497 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:52.984849 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:52.984812 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:53.683983 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:53.683948 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:54.552121 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:54.552070 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:55.295711 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:55.295670 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:55.973848 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:55.973814 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:56.773483 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:56.773450 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:57.571621 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:57.571584 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:58.275198 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:58.275164 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:58.975197 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:58.975158 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:59.260391 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:59.260309 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:59.504760 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:59.504715 5801 scope.go:122] "RemoveContainer" containerID="f16f9bd53e577c6a405956985bb96c061ca424eaa387852c79291ecd03bdb4e0" Oct 30 22:10:59.681726 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:10:59.681693 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:00.278097 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:11:00.278061 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:11:00.387843 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:00.387805 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:01.111719 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:01.111690 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:01.798246 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:01.798209 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:02.497040 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:02.497007 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:03.181224 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:03.181191 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:03.886141 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:03.886108 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:04.651001 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:04.650961 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:05.379617 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:05.379586 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:05.988682 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:05.988645 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:06.676365 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:06.676330 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:07.386080 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:07.386045 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:08.072344 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:08.072305 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:08.875859 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:08.875821 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:09.775824 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:09.775781 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:10.477174 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:10.477126 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:11.027490 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:11.027419 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kb4qt"] Oct 30 22:11:11.029721 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:11.029698 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kb4qt"] Oct 30 22:11:11.178150 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:11.178101 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:11.523082 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:11.523051 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03dcbb5f-dd7f-4ab6-ad37-4452802ea050" path="/var/lib/kubelet/pods/03dcbb5f-dd7f-4ab6-ad37-4452802ea050/volumes" Oct 30 22:11:12.023128 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:12.023093 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:12.768219 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:12.768172 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:13.481973 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:13.481944 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:13.584621 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:13.584585 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:14.176385 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:14.176344 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:14.956277 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:14.956237 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:15.665046 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:15.665012 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:16.383352 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:16.383318 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:17.099084 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:17.099049 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:17.856870 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:17.856833 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:18.579005 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:18.578971 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:19.285699 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:19.285669 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:19.982146 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:19.982111 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:20.688407 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:20.688374 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:21.389002 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:21.388963 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:22.118844 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:22.118800 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:22.895051 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:22.895012 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:23.692123 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:23.692087 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:24.378491 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:24.378431 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:25.081082 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:25.081055 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:25.778156 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:25.778118 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:26.557637 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:26.557600 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:27.277378 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:27.277338 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:28.075119 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:28.075079 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:28.785171 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:28.785137 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:29.481268 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:29.481233 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:30.180665 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:30.180628 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:30.277906 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:11:30.277871 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:11:30.879937 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:30.879887 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:31.261905 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:31.261829 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w"] Oct 30 22:11:31.264986 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:31.264968 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w" Oct 30 22:11:31.269583 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:31.269553 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:11:31.270611 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:31.270588 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:11:31.270741 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:31.270632 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:11:31.270741 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:31.270716 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdvj7\"" Oct 30 22:11:31.277180 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:31.277159 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w"] Oct 30 22:11:31.437046 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:31.437015 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwwkp\" (UniqueName: \"kubernetes.io/projected/84696d2a-1481-4baf-8efd-28db0b248b06-kube-api-access-hwwkp\") pod \"maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w\" (UID: \"84696d2a-1481-4baf-8efd-28db0b248b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w" Oct 30 22:11:31.437202 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:31.437156 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84696d2a-1481-4baf-8efd-28db0b248b06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w\" (UID: \"84696d2a-1481-4baf-8efd-28db0b248b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w" Oct 30 22:11:31.437251 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:31.437213 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84696d2a-1481-4baf-8efd-28db0b248b06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w\" (UID: \"84696d2a-1481-4baf-8efd-28db0b248b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w" Oct 30 22:11:31.538253 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:31.538180 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwwkp\" (UniqueName: \"kubernetes.io/projected/84696d2a-1481-4baf-8efd-28db0b248b06-kube-api-access-hwwkp\") pod \"maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w\" (UID: \"84696d2a-1481-4baf-8efd-28db0b248b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w" Oct 30 22:11:31.538253 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:31.538248 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84696d2a-1481-4baf-8efd-28db0b248b06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w\" (UID: \"84696d2a-1481-4baf-8efd-28db0b248b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w" Oct 30 22:11:31.538484 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:31.538271 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84696d2a-1481-4baf-8efd-28db0b248b06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w\" (UID: \"84696d2a-1481-4baf-8efd-28db0b248b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w" Oct 30 22:11:31.538974 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:31.538947 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84696d2a-1481-4baf-8efd-28db0b248b06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w\" (UID: \"84696d2a-1481-4baf-8efd-28db0b248b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w" Oct 30 22:11:31.540596 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:31.540572 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84696d2a-1481-4baf-8efd-28db0b248b06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w\" (UID: \"84696d2a-1481-4baf-8efd-28db0b248b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w" Oct 30 22:11:31.550653 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:31.550629 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwwkp\" (UniqueName: \"kubernetes.io/projected/84696d2a-1481-4baf-8efd-28db0b248b06-kube-api-access-hwwkp\") pod \"maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w\" (UID: \"84696d2a-1481-4baf-8efd-28db0b248b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w" Oct 30 22:11:31.578796 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:31.578777 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w" Oct 30 22:11:31.854360 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:31.854337 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w"] Oct 30 22:11:31.856552 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:11:31.856522 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84696d2a_1481_4baf_8efd_28db0b248b06.slice/crio-bf5233ec601ce163930fb0ac8cde657722bae8160d9fd7e75261f8aba96addc3 WatchSource:0}: Error finding container bf5233ec601ce163930fb0ac8cde657722bae8160d9fd7e75261f8aba96addc3: Status 404 returned error can't find the container with id bf5233ec601ce163930fb0ac8cde657722bae8160d9fd7e75261f8aba96addc3 Oct 30 22:11:31.860713 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:31.860682 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:32.167451 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:32.167425 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w" event={"ID":"84696d2a-1481-4baf-8efd-28db0b248b06","Type":"ContainerStarted","Data":"bf5233ec601ce163930fb0ac8cde657722bae8160d9fd7e75261f8aba96addc3"} Oct 30 22:11:32.585216 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:32.585176 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:33.175166 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:33.175135 5801 generic.go:356] "Generic (PLEG): container finished" podID="84696d2a-1481-4baf-8efd-28db0b248b06" containerID="cdc932295aac3e13f560ae63f9cf6bf13b5d1fd674c5107ead6654d9675af32d" exitCode=0 Oct 30 22:11:33.175357 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:33.175256 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w" event={"ID":"84696d2a-1481-4baf-8efd-28db0b248b06","Type":"ContainerDied","Data":"cdc932295aac3e13f560ae63f9cf6bf13b5d1fd674c5107ead6654d9675af32d"} Oct 30 22:11:33.279816 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:33.279775 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:33.981472 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:33.981436 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:34.307777 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:34.307741 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w" Oct 30 22:11:34.367811 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:34.367779 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84696d2a-1481-4baf-8efd-28db0b248b06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84696d2a-1481-4baf-8efd-28db0b248b06-maas-api-xks-ca\") pod \"84696d2a-1481-4baf-8efd-28db0b248b06\" (UID: \"84696d2a-1481-4baf-8efd-28db0b248b06\") " Oct 30 22:11:34.367972 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:34.367849 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84696d2a-1481-4baf-8efd-28db0b248b06-kube-api-access-hwwkp\" (UniqueName: \"kubernetes.io/projected/84696d2a-1481-4baf-8efd-28db0b248b06-kube-api-access-hwwkp\") pod \"84696d2a-1481-4baf-8efd-28db0b248b06\" (UID: \"84696d2a-1481-4baf-8efd-28db0b248b06\") " Oct 30 22:11:34.367972 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:34.367902 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84696d2a-1481-4baf-8efd-28db0b248b06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84696d2a-1481-4baf-8efd-28db0b248b06-maas-api-openshift-ca\") pod \"84696d2a-1481-4baf-8efd-28db0b248b06\" (UID: \"84696d2a-1481-4baf-8efd-28db0b248b06\") " Oct 30 22:11:34.368321 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:34.368281 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84696d2a-1481-4baf-8efd-28db0b248b06-maas-api-openshift-ca" pod "84696d2a-1481-4baf-8efd-28db0b248b06" (UID: "84696d2a-1481-4baf-8efd-28db0b248b06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:34.369992 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:34.369969 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84696d2a-1481-4baf-8efd-28db0b248b06-maas-api-xks-ca" pod "84696d2a-1481-4baf-8efd-28db0b248b06" (UID: "84696d2a-1481-4baf-8efd-28db0b248b06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:34.370111 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:34.369988 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84696d2a-1481-4baf-8efd-28db0b248b06-kube-api-access-hwwkp" pod "84696d2a-1481-4baf-8efd-28db0b248b06" (UID: "84696d2a-1481-4baf-8efd-28db0b248b06"). InnerVolumeSpecName "kube-api-access-hwwkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:34.468960 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:34.468928 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84696d2a-1481-4baf-8efd-28db0b248b06-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:34.469106 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:34.468963 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwwkp\" (UniqueName: \"kubernetes.io/projected/84696d2a-1481-4baf-8efd-28db0b248b06-kube-api-access-hwwkp\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:34.469106 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:34.468980 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84696d2a-1481-4baf-8efd-28db0b248b06-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:34.781567 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:34.781531 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:35.186129 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:35.186096 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w" Oct 30 22:11:35.186129 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:35.186119 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w" event={"ID":"84696d2a-1481-4baf-8efd-28db0b248b06","Type":"ContainerDied","Data":"bf5233ec601ce163930fb0ac8cde657722bae8160d9fd7e75261f8aba96addc3"} Oct 30 22:11:35.186599 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:35.186166 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf5233ec601ce163930fb0ac8cde657722bae8160d9fd7e75261f8aba96addc3" Oct 30 22:11:35.562657 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:35.562618 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:36.194884 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:36.194848 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:37.100628 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:37.100584 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:37.873116 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:37.873073 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:38.587902 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:38.587863 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:39.305238 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:39.305180 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:40.082783 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:40.082715 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:40.879336 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:40.879302 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:41.383241 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:41.383208 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j"] Oct 30 22:11:41.395501 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:41.395470 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww"] Oct 30 22:11:41.470046 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:41.469998 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:41.569676 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:41.569642 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:46.383874 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:46.383829 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j" podUID="9e5f9638-8e4e-4199-ac2c-23802b71407a" containerName="payload-processing" containerID="cri-o://2763df86e40ba0d17c7be5c16f84bede2c63c7fd3050721ef070c7ef36674ef7" gracePeriod=25 Oct 30 22:11:46.396325 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:46.396280 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww" podUID="9eb5fbb2-783c-4921-a2ad-8edf9bf67a27" containerName="payload-pre-processing" containerID="cri-o://1dda43baf930733f09a9c334e5930f0758b825ccb2cd9d605b2ec1a6308040d1" gracePeriod=25 Oct 30 22:11:46.585264 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:46.585227 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:46.638226 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:46.638162 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j" Oct 30 22:11:46.661460 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:46.661433 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww" Oct 30 22:11:46.672491 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:46.672467 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e5f9638-8e4e-4199-ac2c-23802b71407a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5f9638-8e4e-4199-ac2c-23802b71407a-plugins-config-volume\") pod \"9e5f9638-8e4e-4199-ac2c-23802b71407a\" (UID: \"9e5f9638-8e4e-4199-ac2c-23802b71407a\") " Oct 30 22:11:46.672637 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:46.672593 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5f9638-8e4e-4199-ac2c-23802b71407a-kube-api-access-zbpqx\" (UniqueName: \"kubernetes.io/projected/9e5f9638-8e4e-4199-ac2c-23802b71407a-kube-api-access-zbpqx\") pod \"9e5f9638-8e4e-4199-ac2c-23802b71407a\" (UID: \"9e5f9638-8e4e-4199-ac2c-23802b71407a\") " Oct 30 22:11:46.672837 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:46.672814 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5f9638-8e4e-4199-ac2c-23802b71407a-plugins-config-volume" pod "9e5f9638-8e4e-4199-ac2c-23802b71407a" (UID: "9e5f9638-8e4e-4199-ac2c-23802b71407a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:46.674576 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:46.674553 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5f9638-8e4e-4199-ac2c-23802b71407a-kube-api-access-zbpqx" pod "9e5f9638-8e4e-4199-ac2c-23802b71407a" (UID: "9e5f9638-8e4e-4199-ac2c-23802b71407a"). InnerVolumeSpecName "kube-api-access-zbpqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:46.681743 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:46.681716 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w"] Oct 30 22:11:46.688116 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:46.688087 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:46.691236 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:46.691215 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7766bb-4v65w"] Oct 30 22:11:46.774147 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:46.774115 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eb5fbb2-783c-4921-a2ad-8edf9bf67a27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb5fbb2-783c-4921-a2ad-8edf9bf67a27-plugins-config-volume\") pod \"9eb5fbb2-783c-4921-a2ad-8edf9bf67a27\" (UID: \"9eb5fbb2-783c-4921-a2ad-8edf9bf67a27\") " Oct 30 22:11:46.774346 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:46.774210 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb5fbb2-783c-4921-a2ad-8edf9bf67a27-kube-api-access-67vjj\" (UniqueName: \"kubernetes.io/projected/9eb5fbb2-783c-4921-a2ad-8edf9bf67a27-kube-api-access-67vjj\") pod \"9eb5fbb2-783c-4921-a2ad-8edf9bf67a27\" (UID: \"9eb5fbb2-783c-4921-a2ad-8edf9bf67a27\") " Oct 30 22:11:46.774528 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:46.774498 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbpqx\" (UniqueName: \"kubernetes.io/projected/9e5f9638-8e4e-4199-ac2c-23802b71407a-kube-api-access-zbpqx\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:46.774609 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:46.774533 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5f9638-8e4e-4199-ac2c-23802b71407a-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:46.774609 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:46.774570 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eb5fbb2-783c-4921-a2ad-8edf9bf67a27-plugins-config-volume" pod "9eb5fbb2-783c-4921-a2ad-8edf9bf67a27" (UID: "9eb5fbb2-783c-4921-a2ad-8edf9bf67a27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:46.776376 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:46.776349 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb5fbb2-783c-4921-a2ad-8edf9bf67a27-kube-api-access-67vjj" pod "9eb5fbb2-783c-4921-a2ad-8edf9bf67a27" (UID: "9eb5fbb2-783c-4921-a2ad-8edf9bf67a27"). InnerVolumeSpecName "kube-api-access-67vjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:46.875776 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:46.875722 5801 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb5fbb2-783c-4921-a2ad-8edf9bf67a27-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:46.875776 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:46.875778 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67vjj\" (UniqueName: \"kubernetes.io/projected/9eb5fbb2-783c-4921-a2ad-8edf9bf67a27-kube-api-access-67vjj\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:47.240448 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:47.240408 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:47.248018 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:47.247989 5801 generic.go:356] "Generic (PLEG): container finished" podID="9eb5fbb2-783c-4921-a2ad-8edf9bf67a27" containerID="1dda43baf930733f09a9c334e5930f0758b825ccb2cd9d605b2ec1a6308040d1" exitCode=0 Oct 30 22:11:47.248173 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:47.248035 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww" Oct 30 22:11:47.248173 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:47.248073 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww" event={"ID":"9eb5fbb2-783c-4921-a2ad-8edf9bf67a27","Type":"ContainerDied","Data":"1dda43baf930733f09a9c334e5930f0758b825ccb2cd9d605b2ec1a6308040d1"} Oct 30 22:11:47.248173 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:47.248125 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww" event={"ID":"9eb5fbb2-783c-4921-a2ad-8edf9bf67a27","Type":"ContainerDied","Data":"923f558f86e2d37965936b0812303ce4c97440c99acbea634d2e3d40b91db6e6"} Oct 30 22:11:47.248173 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:47.248149 5801 scope.go:122] "RemoveContainer" containerID="1dda43baf930733f09a9c334e5930f0758b825ccb2cd9d605b2ec1a6308040d1" Oct 30 22:11:47.249564 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:47.249536 5801 generic.go:356] "Generic (PLEG): container finished" podID="9e5f9638-8e4e-4199-ac2c-23802b71407a" containerID="2763df86e40ba0d17c7be5c16f84bede2c63c7fd3050721ef070c7ef36674ef7" exitCode=0 Oct 30 22:11:47.249695 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:47.249585 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j" Oct 30 22:11:47.249695 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:47.249603 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j" event={"ID":"9e5f9638-8e4e-4199-ac2c-23802b71407a","Type":"ContainerDied","Data":"2763df86e40ba0d17c7be5c16f84bede2c63c7fd3050721ef070c7ef36674ef7"} Oct 30 22:11:47.249695 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:47.249643 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j" event={"ID":"9e5f9638-8e4e-4199-ac2c-23802b71407a","Type":"ContainerDied","Data":"714eef8ffc6bb350b9c333fdf964341f572cfc1e9dba6145d1543f54cd9736b5"} Oct 30 22:11:47.257728 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:47.257708 5801 scope.go:122] "RemoveContainer" containerID="1dda43baf930733f09a9c334e5930f0758b825ccb2cd9d605b2ec1a6308040d1" Oct 30 22:11:47.258011 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:11:47.257992 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dda43baf930733f09a9c334e5930f0758b825ccb2cd9d605b2ec1a6308040d1\": container with ID starting with 1dda43baf930733f09a9c334e5930f0758b825ccb2cd9d605b2ec1a6308040d1 not found: ID does not exist" containerID="1dda43baf930733f09a9c334e5930f0758b825ccb2cd9d605b2ec1a6308040d1" Oct 30 22:11:47.258102 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:47.258017 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dda43baf930733f09a9c334e5930f0758b825ccb2cd9d605b2ec1a6308040d1"} err="failed to get container status \"1dda43baf930733f09a9c334e5930f0758b825ccb2cd9d605b2ec1a6308040d1\": rpc error: code = NotFound desc = could not find container \"1dda43baf930733f09a9c334e5930f0758b825ccb2cd9d605b2ec1a6308040d1\": container with ID starting with 1dda43baf930733f09a9c334e5930f0758b825ccb2cd9d605b2ec1a6308040d1 not found: ID does not exist" Oct 30 22:11:47.258102 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:47.258033 5801 scope.go:122] "RemoveContainer" containerID="2763df86e40ba0d17c7be5c16f84bede2c63c7fd3050721ef070c7ef36674ef7" Oct 30 22:11:47.265994 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:47.265979 5801 scope.go:122] "RemoveContainer" containerID="2763df86e40ba0d17c7be5c16f84bede2c63c7fd3050721ef070c7ef36674ef7" Oct 30 22:11:47.266251 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:11:47.266236 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2763df86e40ba0d17c7be5c16f84bede2c63c7fd3050721ef070c7ef36674ef7\": container with ID starting with 2763df86e40ba0d17c7be5c16f84bede2c63c7fd3050721ef070c7ef36674ef7 not found: ID does not exist" containerID="2763df86e40ba0d17c7be5c16f84bede2c63c7fd3050721ef070c7ef36674ef7" Oct 30 22:11:47.266297 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:47.266257 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2763df86e40ba0d17c7be5c16f84bede2c63c7fd3050721ef070c7ef36674ef7"} err="failed to get container status \"2763df86e40ba0d17c7be5c16f84bede2c63c7fd3050721ef070c7ef36674ef7\": rpc error: code = NotFound desc = could not find container \"2763df86e40ba0d17c7be5c16f84bede2c63c7fd3050721ef070c7ef36674ef7\": container with ID starting with 2763df86e40ba0d17c7be5c16f84bede2c63c7fd3050721ef070c7ef36674ef7 not found: ID does not exist" Oct 30 22:11:47.274396 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:47.274372 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j"] Oct 30 22:11:47.281196 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:47.281170 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-7766bb-d76bff8c5-d2k6j"] Oct 30 22:11:47.295210 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:47.295191 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww"] Oct 30 22:11:47.303054 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:47.303031 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-7766bb-7b97586d8c-xk6ww"] Oct 30 22:11:47.523060 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:47.522975 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84696d2a-1481-4baf-8efd-28db0b248b06" path="/var/lib/kubelet/pods/84696d2a-1481-4baf-8efd-28db0b248b06/volumes" Oct 30 22:11:47.523552 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:47.523486 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e5f9638-8e4e-4199-ac2c-23802b71407a" path="/var/lib/kubelet/pods/9e5f9638-8e4e-4199-ac2c-23802b71407a/volumes" Oct 30 22:11:47.523911 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:47.523891 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eb5fbb2-783c-4921-a2ad-8edf9bf67a27" path="/var/lib/kubelet/pods/9eb5fbb2-783c-4921-a2ad-8edf9bf67a27/volumes" Oct 30 22:11:53.257161 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:53.257124 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:53.367053 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:53.367019 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:53.580177 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:53.580140 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl"] Oct 30 22:11:53.580422 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:53.580393 5801 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" podUID="e81f465f-d0b9-48c0-a7cc-21acb683c890" containerName="istio-proxy" containerID="cri-o://7b2fc290dc2edb0451dd80619eba48c1c5b8555c0f79d623e36f088ba2e95257" gracePeriod=30 Oct 30 22:11:53.623315 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:53.623280 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:58.864724 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:58.864665 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:11:58.927448 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:58.927413 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e81f465f-d0b9-48c0-a7cc-21acb683c890-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e81f465f-d0b9-48c0-a7cc-21acb683c890-istiod-ca-cert\") pod \"e81f465f-d0b9-48c0-a7cc-21acb683c890\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " Oct 30 22:11:58.927610 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:58.927496 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-workload-socket\") pod \"e81f465f-d0b9-48c0-a7cc-21acb683c890\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " Oct 30 22:11:58.927610 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:58.927541 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-workload-certs\") pod \"e81f465f-d0b9-48c0-a7cc-21acb683c890\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " Oct 30 22:11:58.927610 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:58.927571 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-podinfo\") pod \"e81f465f-d0b9-48c0-a7cc-21acb683c890\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " Oct 30 22:11:58.927610 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:58.927592 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e81f465f-d0b9-48c0-a7cc-21acb683c890-kube-api-access-ltk22\" (UniqueName: \"kubernetes.io/projected/e81f465f-d0b9-48c0-a7cc-21acb683c890-kube-api-access-ltk22\") pod \"e81f465f-d0b9-48c0-a7cc-21acb683c890\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " Oct 30 22:11:58.927824 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:58.927620 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-envoy\") pod \"e81f465f-d0b9-48c0-a7cc-21acb683c890\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " Oct 30 22:11:58.927824 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:58.927650 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-credential-socket\") pod \"e81f465f-d0b9-48c0-a7cc-21acb683c890\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " Oct 30 22:11:58.927824 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:58.927697 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-data\") pod \"e81f465f-d0b9-48c0-a7cc-21acb683c890\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " Oct 30 22:11:58.927824 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:58.927736 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-token\" (UniqueName: \"kubernetes.io/projected/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-token\") pod \"e81f465f-d0b9-48c0-a7cc-21acb683c890\" (UID: \"e81f465f-d0b9-48c0-a7cc-21acb683c890\") " Oct 30 22:11:58.928043 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:58.927890 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-workload-socket" pod "e81f465f-d0b9-48c0-a7cc-21acb683c890" (UID: "e81f465f-d0b9-48c0-a7cc-21acb683c890"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:58.928043 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:58.927903 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-workload-certs" pod "e81f465f-d0b9-48c0-a7cc-21acb683c890" (UID: "e81f465f-d0b9-48c0-a7cc-21acb683c890"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:58.928043 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:58.927841 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e81f465f-d0b9-48c0-a7cc-21acb683c890-istiod-ca-cert" pod "e81f465f-d0b9-48c0-a7cc-21acb683c890" (UID: "e81f465f-d0b9-48c0-a7cc-21acb683c890"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:58.928226 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:58.928094 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-data" pod "e81f465f-d0b9-48c0-a7cc-21acb683c890" (UID: "e81f465f-d0b9-48c0-a7cc-21acb683c890"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:58.928226 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:58.928150 5801 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e81f465f-d0b9-48c0-a7cc-21acb683c890-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:58.928226 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:58.928166 5801 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:58.928226 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:58.928176 5801 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:58.928435 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:58.928285 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-credential-socket" pod "e81f465f-d0b9-48c0-a7cc-21acb683c890" (UID: "e81f465f-d0b9-48c0-a7cc-21acb683c890"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:58.930027 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:58.930008 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-envoy" pod "e81f465f-d0b9-48c0-a7cc-21acb683c890" (UID: "e81f465f-d0b9-48c0-a7cc-21acb683c890"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:58.930107 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:58.930010 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-token" pod "e81f465f-d0b9-48c0-a7cc-21acb683c890" (UID: "e81f465f-d0b9-48c0-a7cc-21acb683c890"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:58.930264 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:58.930243 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81f465f-d0b9-48c0-a7cc-21acb683c890-kube-api-access-ltk22" pod "e81f465f-d0b9-48c0-a7cc-21acb683c890" (UID: "e81f465f-d0b9-48c0-a7cc-21acb683c890"). InnerVolumeSpecName "kube-api-access-ltk22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:58.930312 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:58.930260 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-podinfo" pod "e81f465f-d0b9-48c0-a7cc-21acb683c890" (UID: "e81f465f-d0b9-48c0-a7cc-21acb683c890"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:11:59.029018 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:59.028983 5801 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:59.029018 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:59.029013 5801 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:59.029018 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:59.029023 5801 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:59.029278 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:59.029031 5801 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:59.029278 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:59.029042 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltk22\" (UniqueName: \"kubernetes.io/projected/e81f465f-d0b9-48c0-a7cc-21acb683c890-kube-api-access-ltk22\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:59.029278 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:59.029052 5801 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e81f465f-d0b9-48c0-a7cc-21acb683c890-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:59.322352 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:59.322308 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:59.352405 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:59.352376 5801 generic.go:356] "Generic (PLEG): container finished" podID="e81f465f-d0b9-48c0-a7cc-21acb683c890" containerID="7b2fc290dc2edb0451dd80619eba48c1c5b8555c0f79d623e36f088ba2e95257" exitCode=0 Oct 30 22:11:59.352577 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:59.352420 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" Oct 30 22:11:59.352577 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:59.352452 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" event={"ID":"e81f465f-d0b9-48c0-a7cc-21acb683c890","Type":"ContainerDied","Data":"7b2fc290dc2edb0451dd80619eba48c1c5b8555c0f79d623e36f088ba2e95257"} Oct 30 22:11:59.352577 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:59.352505 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl" event={"ID":"e81f465f-d0b9-48c0-a7cc-21acb683c890","Type":"ContainerDied","Data":"e3455978a47d9abf13101702af7ffd1510bbd94787b408bdc4398d7d81c55fde"} Oct 30 22:11:59.352577 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:59.352528 5801 scope.go:122] "RemoveContainer" containerID="7b2fc290dc2edb0451dd80619eba48c1c5b8555c0f79d623e36f088ba2e95257" Oct 30 22:11:59.362120 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:59.362103 5801 scope.go:122] "RemoveContainer" containerID="c2dd91de765143b7bbce2df6fa759330d7409f960d69fda6023b8c0848dcd2fe" Oct 30 22:11:59.370887 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:59.370860 5801 scope.go:122] "RemoveContainer" containerID="7b2fc290dc2edb0451dd80619eba48c1c5b8555c0f79d623e36f088ba2e95257" Oct 30 22:11:59.371121 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:11:59.371099 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b2fc290dc2edb0451dd80619eba48c1c5b8555c0f79d623e36f088ba2e95257\": container with ID starting with 7b2fc290dc2edb0451dd80619eba48c1c5b8555c0f79d623e36f088ba2e95257 not found: ID does not exist" containerID="7b2fc290dc2edb0451dd80619eba48c1c5b8555c0f79d623e36f088ba2e95257" Oct 30 22:11:59.371211 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:59.371133 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b2fc290dc2edb0451dd80619eba48c1c5b8555c0f79d623e36f088ba2e95257"} err="failed to get container status \"7b2fc290dc2edb0451dd80619eba48c1c5b8555c0f79d623e36f088ba2e95257\": rpc error: code = NotFound desc = could not find container \"7b2fc290dc2edb0451dd80619eba48c1c5b8555c0f79d623e36f088ba2e95257\": container with ID starting with 7b2fc290dc2edb0451dd80619eba48c1c5b8555c0f79d623e36f088ba2e95257 not found: ID does not exist" Oct 30 22:11:59.371211 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:59.371157 5801 scope.go:122] "RemoveContainer" containerID="c2dd91de765143b7bbce2df6fa759330d7409f960d69fda6023b8c0848dcd2fe" Oct 30 22:11:59.371379 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:11:59.371362 5801 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2dd91de765143b7bbce2df6fa759330d7409f960d69fda6023b8c0848dcd2fe\": container with ID starting with c2dd91de765143b7bbce2df6fa759330d7409f960d69fda6023b8c0848dcd2fe not found: ID does not exist" containerID="c2dd91de765143b7bbce2df6fa759330d7409f960d69fda6023b8c0848dcd2fe" Oct 30 22:11:59.371438 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:59.371387 5801 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2dd91de765143b7bbce2df6fa759330d7409f960d69fda6023b8c0848dcd2fe"} err="failed to get container status \"c2dd91de765143b7bbce2df6fa759330d7409f960d69fda6023b8c0848dcd2fe\": rpc error: code = NotFound desc = could not find container \"c2dd91de765143b7bbce2df6fa759330d7409f960d69fda6023b8c0848dcd2fe\": container with ID starting with c2dd91de765143b7bbce2df6fa759330d7409f960d69fda6023b8c0848dcd2fe not found: ID does not exist" Oct 30 22:11:59.376743 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:59.376717 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl"] Oct 30 22:11:59.381735 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:59.381717 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-7766bb-openshift-default-846b5d9665-7bdtl"] Oct 30 22:11:59.523078 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:59.523050 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e81f465f-d0b9-48c0-a7cc-21acb683c890" path="/var/lib/kubelet/pods/e81f465f-d0b9-48c0-a7cc-21acb683c890/volumes" Oct 30 22:11:59.524902 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:11:59.524875 5801 scope.go:122] "RemoveContainer" containerID="515444b3db68347541f6a85f1a4bdc88025a1514ac81dea7d8984feda874d066" Oct 30 22:12:00.277680 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:12:00.277642 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:12:05.100688 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.100652 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jv74x"] Oct 30 22:12:05.101169 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.101013 5801 cpu_manager.go:414] "containerMap: removing container" podUID="e81f465f-d0b9-48c0-a7cc-21acb683c890" containerName="istio-proxy" Oct 30 22:12:05.101169 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.101024 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e81f465f-d0b9-48c0-a7cc-21acb683c890" containerName="istio-proxy" Oct 30 22:12:05.101169 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.101039 5801 cpu_manager.go:414] "containerMap: removing container" podUID="9e5f9638-8e4e-4199-ac2c-23802b71407a" containerName="payload-processing" Oct 30 22:12:05.101169 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.101044 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5f9638-8e4e-4199-ac2c-23802b71407a" containerName="payload-processing" Oct 30 22:12:05.101169 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.101052 5801 cpu_manager.go:414] "containerMap: removing container" podUID="9eb5fbb2-783c-4921-a2ad-8edf9bf67a27" containerName="payload-pre-processing" Oct 30 22:12:05.101169 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.101057 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb5fbb2-783c-4921-a2ad-8edf9bf67a27" containerName="payload-pre-processing" Oct 30 22:12:05.101169 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.101071 5801 cpu_manager.go:414] "containerMap: removing container" podUID="84696d2a-1481-4baf-8efd-28db0b248b06" containerName="revoke-keys" Oct 30 22:12:05.101169 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.101076 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84696d2a-1481-4baf-8efd-28db0b248b06" containerName="revoke-keys" Oct 30 22:12:05.101169 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.101141 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="e81f465f-d0b9-48c0-a7cc-21acb683c890" containerName="istio-proxy" Oct 30 22:12:05.101169 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.101152 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5f9638-8e4e-4199-ac2c-23802b71407a" containerName="payload-processing" Oct 30 22:12:05.101169 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.101158 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb5fbb2-783c-4921-a2ad-8edf9bf67a27" containerName="payload-pre-processing" Oct 30 22:12:05.101169 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.101166 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="e81f465f-d0b9-48c0-a7cc-21acb683c890" containerName="istio-proxy" Oct 30 22:12:05.101169 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.101172 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="84696d2a-1481-4baf-8efd-28db0b248b06" containerName="revoke-keys" Oct 30 22:12:05.105817 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.105800 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jv74x" Oct 30 22:12:05.112041 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.112018 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:12:05.113386 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.113126 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:12:05.113386 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.113153 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:12:05.113386 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.113250 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdvj7\"" Oct 30 22:12:05.116430 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.116192 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jv74x"] Oct 30 22:12:05.260010 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.259971 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5230b38d-7e4a-4cb3-8071-6f3484fd4240-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-jv74x\" (UID: \"5230b38d-7e4a-4cb3-8071-6f3484fd4240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jv74x" Oct 30 22:12:05.260186 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.260042 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp5sl\" (UniqueName: \"kubernetes.io/projected/5230b38d-7e4a-4cb3-8071-6f3484fd4240-kube-api-access-kp5sl\") pod \"maas-api-revoke-sub-f03629db-jv74x\" (UID: \"5230b38d-7e4a-4cb3-8071-6f3484fd4240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jv74x" Oct 30 22:12:05.260186 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.260130 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5230b38d-7e4a-4cb3-8071-6f3484fd4240-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-jv74x\" (UID: \"5230b38d-7e4a-4cb3-8071-6f3484fd4240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jv74x" Oct 30 22:12:05.360783 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.360665 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5230b38d-7e4a-4cb3-8071-6f3484fd4240-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-jv74x\" (UID: \"5230b38d-7e4a-4cb3-8071-6f3484fd4240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jv74x" Oct 30 22:12:05.360934 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.360789 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5230b38d-7e4a-4cb3-8071-6f3484fd4240-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-jv74x\" (UID: \"5230b38d-7e4a-4cb3-8071-6f3484fd4240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jv74x" Oct 30 22:12:05.360934 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.360867 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp5sl\" (UniqueName: \"kubernetes.io/projected/5230b38d-7e4a-4cb3-8071-6f3484fd4240-kube-api-access-kp5sl\") pod \"maas-api-revoke-sub-f03629db-jv74x\" (UID: \"5230b38d-7e4a-4cb3-8071-6f3484fd4240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jv74x" Oct 30 22:12:05.361586 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.361546 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5230b38d-7e4a-4cb3-8071-6f3484fd4240-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-jv74x\" (UID: \"5230b38d-7e4a-4cb3-8071-6f3484fd4240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jv74x" Oct 30 22:12:05.363511 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.363479 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5230b38d-7e4a-4cb3-8071-6f3484fd4240-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-jv74x\" (UID: \"5230b38d-7e4a-4cb3-8071-6f3484fd4240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jv74x" Oct 30 22:12:05.373967 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.373944 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp5sl\" (UniqueName: \"kubernetes.io/projected/5230b38d-7e4a-4cb3-8071-6f3484fd4240-kube-api-access-kp5sl\") pod \"maas-api-revoke-sub-f03629db-jv74x\" (UID: \"5230b38d-7e4a-4cb3-8071-6f3484fd4240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jv74x" Oct 30 22:12:05.420158 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.420135 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jv74x" Oct 30 22:12:05.693499 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:05.693466 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jv74x"] Oct 30 22:12:05.696110 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:12:05.696081 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5230b38d_7e4a_4cb3_8071_6f3484fd4240.slice/crio-d076ebf995cd3a26ceb829a745436e8b515e5ecf18d2b85f69ddd1345e3242a5 WatchSource:0}: Error finding container d076ebf995cd3a26ceb829a745436e8b515e5ecf18d2b85f69ddd1345e3242a5: Status 404 returned error can't find the container with id d076ebf995cd3a26ceb829a745436e8b515e5ecf18d2b85f69ddd1345e3242a5 Oct 30 22:12:06.416204 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:06.416171 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jv74x" event={"ID":"5230b38d-7e4a-4cb3-8071-6f3484fd4240","Type":"ContainerStarted","Data":"31f3e7b7b987d1ccbe0e206228c184171cd163cb365a23e5560251cc880d4811"} Oct 30 22:12:06.416583 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:06.416221 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jv74x" event={"ID":"5230b38d-7e4a-4cb3-8071-6f3484fd4240","Type":"ContainerStarted","Data":"d076ebf995cd3a26ceb829a745436e8b515e5ecf18d2b85f69ddd1345e3242a5"} Oct 30 22:12:06.437029 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:06.436978 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jv74x" podStartSLOduration=1.4369662650000001 podStartE2EDuration="1.436966265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:06.434501989 +0000 UTC m=+2595.030318660" watchObservedRunningTime="2026-10-30 22:12:06.436966265 +0000 UTC m=+2595.032782936" Oct 30 22:12:07.422646 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:07.422608 5801 generic.go:356] "Generic (PLEG): container finished" podID="5230b38d-7e4a-4cb3-8071-6f3484fd4240" containerID="31f3e7b7b987d1ccbe0e206228c184171cd163cb365a23e5560251cc880d4811" exitCode=0 Oct 30 22:12:07.423084 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:07.422696 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jv74x" event={"ID":"5230b38d-7e4a-4cb3-8071-6f3484fd4240","Type":"ContainerDied","Data":"31f3e7b7b987d1ccbe0e206228c184171cd163cb365a23e5560251cc880d4811"} Oct 30 22:12:08.558929 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:08.558902 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jv74x" Oct 30 22:12:08.692704 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:08.692622 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5230b38d-7e4a-4cb3-8071-6f3484fd4240-kube-api-access-kp5sl\" (UniqueName: \"kubernetes.io/projected/5230b38d-7e4a-4cb3-8071-6f3484fd4240-kube-api-access-kp5sl\") pod \"5230b38d-7e4a-4cb3-8071-6f3484fd4240\" (UID: \"5230b38d-7e4a-4cb3-8071-6f3484fd4240\") " Oct 30 22:12:08.692937 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:08.692727 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5230b38d-7e4a-4cb3-8071-6f3484fd4240-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5230b38d-7e4a-4cb3-8071-6f3484fd4240-maas-api-xks-ca\") pod \"5230b38d-7e4a-4cb3-8071-6f3484fd4240\" (UID: \"5230b38d-7e4a-4cb3-8071-6f3484fd4240\") " Oct 30 22:12:08.692937 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:08.692748 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5230b38d-7e4a-4cb3-8071-6f3484fd4240-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5230b38d-7e4a-4cb3-8071-6f3484fd4240-maas-api-openshift-ca\") pod \"5230b38d-7e4a-4cb3-8071-6f3484fd4240\" (UID: \"5230b38d-7e4a-4cb3-8071-6f3484fd4240\") " Oct 30 22:12:08.693168 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:08.693144 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5230b38d-7e4a-4cb3-8071-6f3484fd4240-maas-api-openshift-ca" pod "5230b38d-7e4a-4cb3-8071-6f3484fd4240" (UID: "5230b38d-7e4a-4cb3-8071-6f3484fd4240"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:08.694796 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:08.694774 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5230b38d-7e4a-4cb3-8071-6f3484fd4240-maas-api-xks-ca" pod "5230b38d-7e4a-4cb3-8071-6f3484fd4240" (UID: "5230b38d-7e4a-4cb3-8071-6f3484fd4240"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:08.694917 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:08.694900 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5230b38d-7e4a-4cb3-8071-6f3484fd4240-kube-api-access-kp5sl" pod "5230b38d-7e4a-4cb3-8071-6f3484fd4240" (UID: "5230b38d-7e4a-4cb3-8071-6f3484fd4240"). InnerVolumeSpecName "kube-api-access-kp5sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:08.793474 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:08.793444 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5230b38d-7e4a-4cb3-8071-6f3484fd4240-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:08.793474 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:08.793471 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5230b38d-7e4a-4cb3-8071-6f3484fd4240-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:08.793474 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:08.793482 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp5sl\" (UniqueName: \"kubernetes.io/projected/5230b38d-7e4a-4cb3-8071-6f3484fd4240-kube-api-access-kp5sl\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:09.434536 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:09.434495 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jv74x" Oct 30 22:12:09.434721 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:09.434500 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jv74x" event={"ID":"5230b38d-7e4a-4cb3-8071-6f3484fd4240","Type":"ContainerDied","Data":"d076ebf995cd3a26ceb829a745436e8b515e5ecf18d2b85f69ddd1345e3242a5"} Oct 30 22:12:09.434721 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:09.434628 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d076ebf995cd3a26ceb829a745436e8b515e5ecf18d2b85f69ddd1345e3242a5" Oct 30 22:12:15.029370 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:15.029329 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d98392e-hm2tb"] Oct 30 22:12:15.033971 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:15.033942 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d98392e-hm2tb"] Oct 30 22:12:15.522628 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:15.522593 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="467e3901-d46d-4721-a24c-ee97d242e806" path="/var/lib/kubelet/pods/467e3901-d46d-4721-a24c-ee97d242e806/volumes" Oct 30 22:12:21.147648 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:21.147614 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-b92qp"] Oct 30 22:12:21.148152 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:21.148133 5801 cpu_manager.go:414] "containerMap: removing container" podUID="5230b38d-7e4a-4cb3-8071-6f3484fd4240" containerName="revoke-keys" Oct 30 22:12:21.148212 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:21.148154 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5230b38d-7e4a-4cb3-8071-6f3484fd4240" containerName="revoke-keys" Oct 30 22:12:21.148212 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:21.148167 5801 cpu_manager.go:414] "containerMap: removing container" podUID="e81f465f-d0b9-48c0-a7cc-21acb683c890" containerName="istio-proxy" Oct 30 22:12:21.148212 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:21.148173 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e81f465f-d0b9-48c0-a7cc-21acb683c890" containerName="istio-proxy" Oct 30 22:12:21.148338 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:21.148242 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="5230b38d-7e4a-4cb3-8071-6f3484fd4240" containerName="revoke-keys" Oct 30 22:12:21.151132 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:21.151116 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-b92qp" Oct 30 22:12:21.154997 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:21.154962 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:12:21.154997 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:21.154970 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:12:21.156061 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:21.156037 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:12:21.156061 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:21.156057 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdvj7\"" Oct 30 22:12:21.162640 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:21.162616 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-b92qp"] Oct 30 22:12:21.179037 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:21.179014 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011bdd2c-2993-403f-a45d-443f2fa44ce2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-b92qp\" (UID: \"011bdd2c-2993-403f-a45d-443f2fa44ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-b92qp" Oct 30 22:12:21.179178 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:21.179088 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011bdd2c-2993-403f-a45d-443f2fa44ce2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-b92qp\" (UID: \"011bdd2c-2993-403f-a45d-443f2fa44ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-b92qp" Oct 30 22:12:21.179178 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:21.179143 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzbzt\" (UniqueName: \"kubernetes.io/projected/011bdd2c-2993-403f-a45d-443f2fa44ce2-kube-api-access-wzbzt\") pod \"maas-api-revoke-sub-0165d7d2-b92qp\" (UID: \"011bdd2c-2993-403f-a45d-443f2fa44ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-b92qp" Oct 30 22:12:21.280010 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:21.279979 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011bdd2c-2993-403f-a45d-443f2fa44ce2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-b92qp\" (UID: \"011bdd2c-2993-403f-a45d-443f2fa44ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-b92qp" Oct 30 22:12:21.280162 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:21.280059 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzbzt\" (UniqueName: \"kubernetes.io/projected/011bdd2c-2993-403f-a45d-443f2fa44ce2-kube-api-access-wzbzt\") pod \"maas-api-revoke-sub-0165d7d2-b92qp\" (UID: \"011bdd2c-2993-403f-a45d-443f2fa44ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-b92qp" Oct 30 22:12:21.280162 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:21.280150 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011bdd2c-2993-403f-a45d-443f2fa44ce2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-b92qp\" (UID: \"011bdd2c-2993-403f-a45d-443f2fa44ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-b92qp" Oct 30 22:12:21.280678 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:21.280647 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011bdd2c-2993-403f-a45d-443f2fa44ce2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-b92qp\" (UID: \"011bdd2c-2993-403f-a45d-443f2fa44ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-b92qp" Oct 30 22:12:21.282503 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:21.282484 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011bdd2c-2993-403f-a45d-443f2fa44ce2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-b92qp\" (UID: \"011bdd2c-2993-403f-a45d-443f2fa44ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-b92qp" Oct 30 22:12:21.292994 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:21.292968 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzbzt\" (UniqueName: \"kubernetes.io/projected/011bdd2c-2993-403f-a45d-443f2fa44ce2-kube-api-access-wzbzt\") pod \"maas-api-revoke-sub-0165d7d2-b92qp\" (UID: \"011bdd2c-2993-403f-a45d-443f2fa44ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-b92qp" Oct 30 22:12:21.464870 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:21.464793 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-b92qp" Oct 30 22:12:21.776526 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:21.776502 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-b92qp"] Oct 30 22:12:21.779527 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:12:21.779498 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011bdd2c_2993_403f_a45d_443f2fa44ce2.slice/crio-f95585a0818dfa7d5bb0c3adf35a8f229ae967131f2742d1aaf1943032d295c4 WatchSource:0}: Error finding container f95585a0818dfa7d5bb0c3adf35a8f229ae967131f2742d1aaf1943032d295c4: Status 404 returned error can't find the container with id f95585a0818dfa7d5bb0c3adf35a8f229ae967131f2742d1aaf1943032d295c4 Oct 30 22:12:22.522378 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:22.522281 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-b92qp" event={"ID":"011bdd2c-2993-403f-a45d-443f2fa44ce2","Type":"ContainerStarted","Data":"6fbbb94ef729eedfc4eb4b512f3ac9fd755f2bcc2817b6214b935912c11dc582"} Oct 30 22:12:22.522378 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:22.522341 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-b92qp" event={"ID":"011bdd2c-2993-403f-a45d-443f2fa44ce2","Type":"ContainerStarted","Data":"f95585a0818dfa7d5bb0c3adf35a8f229ae967131f2742d1aaf1943032d295c4"} Oct 30 22:12:22.543118 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:22.543070 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-b92qp" podStartSLOduration=1.5430578289999999 podStartE2EDuration="1.543057829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:22.54038985 +0000 UTC m=+2611.136206521" watchObservedRunningTime="2026-10-30 22:12:22.543057829 +0000 UTC m=+2611.138874483" Oct 30 22:12:23.528290 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:23.528255 5801 generic.go:356] "Generic (PLEG): container finished" podID="011bdd2c-2993-403f-a45d-443f2fa44ce2" containerID="6fbbb94ef729eedfc4eb4b512f3ac9fd755f2bcc2817b6214b935912c11dc582" exitCode=0 Oct 30 22:12:23.528689 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:23.528312 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-b92qp" event={"ID":"011bdd2c-2993-403f-a45d-443f2fa44ce2","Type":"ContainerDied","Data":"6fbbb94ef729eedfc4eb4b512f3ac9fd755f2bcc2817b6214b935912c11dc582"} Oct 30 22:12:24.655503 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:24.655478 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-b92qp" Oct 30 22:12:24.706104 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:24.706078 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/011bdd2c-2993-403f-a45d-443f2fa44ce2-kube-api-access-wzbzt\" (UniqueName: \"kubernetes.io/projected/011bdd2c-2993-403f-a45d-443f2fa44ce2-kube-api-access-wzbzt\") pod \"011bdd2c-2993-403f-a45d-443f2fa44ce2\" (UID: \"011bdd2c-2993-403f-a45d-443f2fa44ce2\") " Oct 30 22:12:24.706227 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:24.706135 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/011bdd2c-2993-403f-a45d-443f2fa44ce2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011bdd2c-2993-403f-a45d-443f2fa44ce2-maas-api-xks-ca\") pod \"011bdd2c-2993-403f-a45d-443f2fa44ce2\" (UID: \"011bdd2c-2993-403f-a45d-443f2fa44ce2\") " Oct 30 22:12:24.706227 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:24.706199 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/011bdd2c-2993-403f-a45d-443f2fa44ce2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011bdd2c-2993-403f-a45d-443f2fa44ce2-maas-api-openshift-ca\") pod \"011bdd2c-2993-403f-a45d-443f2fa44ce2\" (UID: \"011bdd2c-2993-403f-a45d-443f2fa44ce2\") " Oct 30 22:12:24.706654 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:24.706624 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/011bdd2c-2993-403f-a45d-443f2fa44ce2-maas-api-openshift-ca" pod "011bdd2c-2993-403f-a45d-443f2fa44ce2" (UID: "011bdd2c-2993-403f-a45d-443f2fa44ce2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:24.708100 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:24.708073 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/011bdd2c-2993-403f-a45d-443f2fa44ce2-kube-api-access-wzbzt" pod "011bdd2c-2993-403f-a45d-443f2fa44ce2" (UID: "011bdd2c-2993-403f-a45d-443f2fa44ce2"). InnerVolumeSpecName "kube-api-access-wzbzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:24.708192 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:24.708142 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/011bdd2c-2993-403f-a45d-443f2fa44ce2-maas-api-xks-ca" pod "011bdd2c-2993-403f-a45d-443f2fa44ce2" (UID: "011bdd2c-2993-403f-a45d-443f2fa44ce2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:24.808395 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:24.808368 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011bdd2c-2993-403f-a45d-443f2fa44ce2-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:24.808395 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:24.808394 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzbzt\" (UniqueName: \"kubernetes.io/projected/011bdd2c-2993-403f-a45d-443f2fa44ce2-kube-api-access-wzbzt\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:24.808579 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:24.808407 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011bdd2c-2993-403f-a45d-443f2fa44ce2-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:25.538321 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:25.538295 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-b92qp" Oct 30 22:12:25.538438 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:25.538332 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-b92qp" event={"ID":"011bdd2c-2993-403f-a45d-443f2fa44ce2","Type":"ContainerDied","Data":"f95585a0818dfa7d5bb0c3adf35a8f229ae967131f2742d1aaf1943032d295c4"} Oct 30 22:12:25.538438 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:25.538372 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f95585a0818dfa7d5bb0c3adf35a8f229ae967131f2742d1aaf1943032d295c4" Oct 30 22:12:30.277558 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:12:30.277521 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:12:32.023234 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:32.023187 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xjnf4"] Oct 30 22:12:32.023945 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:32.023916 5801 cpu_manager.go:414] "containerMap: removing container" podUID="011bdd2c-2993-403f-a45d-443f2fa44ce2" containerName="revoke-keys" Oct 30 22:12:32.023996 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:32.023950 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="011bdd2c-2993-403f-a45d-443f2fa44ce2" containerName="revoke-keys" Oct 30 22:12:32.024092 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:32.024076 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="011bdd2c-2993-403f-a45d-443f2fa44ce2" containerName="revoke-keys" Oct 30 22:12:32.027973 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:32.027950 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xjnf4" Oct 30 22:12:32.038390 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:32.038364 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:12:32.038597 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:32.038385 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:12:32.038597 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:32.038532 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:12:32.038873 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:32.038850 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xjnf4"] Oct 30 22:12:32.039344 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:32.039327 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdvj7\"" Oct 30 22:12:32.058273 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:32.058247 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7894eb2b-44ba-4ae5-81dd-ff5192129be2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xjnf4\" (UID: \"7894eb2b-44ba-4ae5-81dd-ff5192129be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xjnf4" Oct 30 22:12:32.058410 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:32.058290 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv6kt\" (UniqueName: \"kubernetes.io/projected/7894eb2b-44ba-4ae5-81dd-ff5192129be2-kube-api-access-fv6kt\") pod \"maas-api-revoke-sub-6b19338f-xjnf4\" (UID: \"7894eb2b-44ba-4ae5-81dd-ff5192129be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xjnf4" Oct 30 22:12:32.058480 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:32.058452 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7894eb2b-44ba-4ae5-81dd-ff5192129be2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xjnf4\" (UID: \"7894eb2b-44ba-4ae5-81dd-ff5192129be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xjnf4" Oct 30 22:12:32.159211 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:32.159166 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7894eb2b-44ba-4ae5-81dd-ff5192129be2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xjnf4\" (UID: \"7894eb2b-44ba-4ae5-81dd-ff5192129be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xjnf4" Oct 30 22:12:32.159211 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:32.159218 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv6kt\" (UniqueName: \"kubernetes.io/projected/7894eb2b-44ba-4ae5-81dd-ff5192129be2-kube-api-access-fv6kt\") pod \"maas-api-revoke-sub-6b19338f-xjnf4\" (UID: \"7894eb2b-44ba-4ae5-81dd-ff5192129be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xjnf4" Oct 30 22:12:32.159454 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:32.159257 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7894eb2b-44ba-4ae5-81dd-ff5192129be2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xjnf4\" (UID: \"7894eb2b-44ba-4ae5-81dd-ff5192129be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xjnf4" Oct 30 22:12:32.159926 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:32.159907 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7894eb2b-44ba-4ae5-81dd-ff5192129be2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xjnf4\" (UID: \"7894eb2b-44ba-4ae5-81dd-ff5192129be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xjnf4" Oct 30 22:12:32.161470 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:32.161450 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7894eb2b-44ba-4ae5-81dd-ff5192129be2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xjnf4\" (UID: \"7894eb2b-44ba-4ae5-81dd-ff5192129be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xjnf4" Oct 30 22:12:32.172641 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:32.172620 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv6kt\" (UniqueName: \"kubernetes.io/projected/7894eb2b-44ba-4ae5-81dd-ff5192129be2-kube-api-access-fv6kt\") pod \"maas-api-revoke-sub-6b19338f-xjnf4\" (UID: \"7894eb2b-44ba-4ae5-81dd-ff5192129be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xjnf4" Oct 30 22:12:32.341683 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:32.341657 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xjnf4" Oct 30 22:12:32.615543 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:32.615517 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xjnf4"] Oct 30 22:12:32.617056 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:12:32.617027 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7894eb2b_44ba_4ae5_81dd_ff5192129be2.slice/crio-bef3b0812fd6d4eece1d0128360638bc218d97e06ca1951fb9a23a22d1bdf528 WatchSource:0}: Error finding container bef3b0812fd6d4eece1d0128360638bc218d97e06ca1951fb9a23a22d1bdf528: Status 404 returned error can't find the container with id bef3b0812fd6d4eece1d0128360638bc218d97e06ca1951fb9a23a22d1bdf528 Oct 30 22:12:33.578975 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:33.578938 5801 generic.go:356] "Generic (PLEG): container finished" podID="7894eb2b-44ba-4ae5-81dd-ff5192129be2" containerID="37c5b186345ebdce1225b9fb0a82470dec220b1be38c284be08b55529bf9491c" exitCode=0 Oct 30 22:12:33.579402 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:33.579027 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xjnf4" event={"ID":"7894eb2b-44ba-4ae5-81dd-ff5192129be2","Type":"ContainerDied","Data":"37c5b186345ebdce1225b9fb0a82470dec220b1be38c284be08b55529bf9491c"} Oct 30 22:12:33.579402 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:33.579128 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xjnf4" event={"ID":"7894eb2b-44ba-4ae5-81dd-ff5192129be2","Type":"ContainerStarted","Data":"bef3b0812fd6d4eece1d0128360638bc218d97e06ca1951fb9a23a22d1bdf528"} Oct 30 22:12:34.713746 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:34.713723 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xjnf4" Oct 30 22:12:34.781811 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:34.781788 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7894eb2b-44ba-4ae5-81dd-ff5192129be2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7894eb2b-44ba-4ae5-81dd-ff5192129be2-maas-api-openshift-ca\") pod \"7894eb2b-44ba-4ae5-81dd-ff5192129be2\" (UID: \"7894eb2b-44ba-4ae5-81dd-ff5192129be2\") " Oct 30 22:12:34.781959 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:34.781837 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7894eb2b-44ba-4ae5-81dd-ff5192129be2-kube-api-access-fv6kt\" (UniqueName: \"kubernetes.io/projected/7894eb2b-44ba-4ae5-81dd-ff5192129be2-kube-api-access-fv6kt\") pod \"7894eb2b-44ba-4ae5-81dd-ff5192129be2\" (UID: \"7894eb2b-44ba-4ae5-81dd-ff5192129be2\") " Oct 30 22:12:34.781959 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:34.781897 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7894eb2b-44ba-4ae5-81dd-ff5192129be2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7894eb2b-44ba-4ae5-81dd-ff5192129be2-maas-api-xks-ca\") pod \"7894eb2b-44ba-4ae5-81dd-ff5192129be2\" (UID: \"7894eb2b-44ba-4ae5-81dd-ff5192129be2\") " Oct 30 22:12:34.782198 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:34.782171 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7894eb2b-44ba-4ae5-81dd-ff5192129be2-maas-api-openshift-ca" pod "7894eb2b-44ba-4ae5-81dd-ff5192129be2" (UID: "7894eb2b-44ba-4ae5-81dd-ff5192129be2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:34.783870 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:34.783850 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7894eb2b-44ba-4ae5-81dd-ff5192129be2-maas-api-xks-ca" pod "7894eb2b-44ba-4ae5-81dd-ff5192129be2" (UID: "7894eb2b-44ba-4ae5-81dd-ff5192129be2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:34.783941 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:34.783910 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7894eb2b-44ba-4ae5-81dd-ff5192129be2-kube-api-access-fv6kt" pod "7894eb2b-44ba-4ae5-81dd-ff5192129be2" (UID: "7894eb2b-44ba-4ae5-81dd-ff5192129be2"). InnerVolumeSpecName "kube-api-access-fv6kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:34.882850 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:34.882797 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv6kt\" (UniqueName: \"kubernetes.io/projected/7894eb2b-44ba-4ae5-81dd-ff5192129be2-kube-api-access-fv6kt\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:34.882850 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:34.882820 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7894eb2b-44ba-4ae5-81dd-ff5192129be2-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:34.882850 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:34.882830 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7894eb2b-44ba-4ae5-81dd-ff5192129be2-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:35.590427 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:35.590403 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xjnf4" Oct 30 22:12:35.590618 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:35.590402 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xjnf4" event={"ID":"7894eb2b-44ba-4ae5-81dd-ff5192129be2","Type":"ContainerDied","Data":"bef3b0812fd6d4eece1d0128360638bc218d97e06ca1951fb9a23a22d1bdf528"} Oct 30 22:12:35.590618 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:35.590534 5801 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bef3b0812fd6d4eece1d0128360638bc218d97e06ca1951fb9a23a22d1bdf528" Oct 30 22:12:56.901428 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:56.901392 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-b92qp_011bdd2c-2993-403f-a45d-443f2fa44ce2/revoke-keys/0.log" Oct 30 22:12:57.080733 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:57.080698 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-c8jdn_f1124264-b8d0-4bde-b039-d3f9f68d5802/revoke-keys/0.log" Oct 30 22:12:57.266209 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:57.266142 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-rxvkm_0f4fb010-f669-42d4-a654-91ca72cc9e82/revoke-keys/0.log" Oct 30 22:12:57.447723 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:57.447691 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-xjnf4_7894eb2b-44ba-4ae5-81dd-ff5192129be2/revoke-keys/0.log" Oct 30 22:12:57.676048 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:57.676016 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-db0fa3aa-5ktnz_137ef9a5-5d59-4c38-8890-fe71531e4d80/revoke-keys/0.log" Oct 30 22:12:57.860626 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:57.860603 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-jv74x_5230b38d-7e4a-4cb3-8071-6f3484fd4240/revoke-keys/0.log" Oct 30 22:12:59.381257 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:59.381216 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:59.559955 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:59.559921 5801 scope.go:122] "RemoveContainer" containerID="e5b15a6702c2cd8418a4ebf3d013ad37043bd7db7a66eb2d2c341e27166343ac" Oct 30 22:12:59.829504 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:12:59.829471 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-fhf5g_fc623bd2-899d-4cfb-ac2b-4f6a8d0bd00d/manager/0.log" Oct 30 22:13:00.277363 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:13:00.277278 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:13:02.169824 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:02.169785 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5978df7f68-prrl7_56bc9daf-0942-46f1-aa51-49b8c868aa77/kube-auth-proxy/0.log" Oct 30 22:13:02.576032 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:02.575985 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-7wxsr_fda6dedb-779c-49d8-8fcf-ba31f73402ae/istio-proxy/0.log" Oct 30 22:13:02.770982 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:02.770950 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-8fqrl_57fe57d2-19a5-4ef7-b606-7753b0b7d902/istio-proxy/0.log" Oct 30 22:13:03.154882 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:03.154855 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-fm6jd_913cd5d1-0d49-41ba-921d-603a44b2126a/istio-proxy/0.log" Oct 30 22:13:03.350380 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:03.350346 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-hqmk4_c1e2cfd5-7f5a-4624-a753-30bc982288d4/istio-proxy/0.log" Oct 30 22:13:04.493746 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:04.493714 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-hm54n_006ecdc9-450d-4058-bd65-bebafca12ea0/payload-pre-processing/0.log" Oct 30 22:13:04.676147 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:04.676119 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-b87ts_06ff0bb8-b3e4-406a-a0b3-b20f27750e47/payload-processing/0.log" Oct 30 22:13:06.049683 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:06.049647 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-6vklf_7e0434b3-6ab0-42c5-9c5b-e4592daadca7/main/0.log" Oct 30 22:13:06.422610 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:06.422579 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv_b6307469-5f7d-41d9-9490-e8381ff9bbfb/main/0.log" Oct 30 22:13:06.774677 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:06.774593 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h_edecde9f-7b29-4563-bf29-bcb748233470/main/0.log" Oct 30 22:13:06.945696 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:06.945662 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6_a8c67e3a-dafa-4ed4-b5e3-f1b341427687/main/0.log" Oct 30 22:13:19.642337 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:19.642305 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-b92qp_011bdd2c-2993-403f-a45d-443f2fa44ce2/revoke-keys/0.log" Oct 30 22:13:19.840604 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:19.840573 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-c8jdn_f1124264-b8d0-4bde-b039-d3f9f68d5802/revoke-keys/0.log" Oct 30 22:13:20.030567 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:20.030477 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-rxvkm_0f4fb010-f669-42d4-a654-91ca72cc9e82/revoke-keys/0.log" Oct 30 22:13:20.224992 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:20.224961 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-xjnf4_7894eb2b-44ba-4ae5-81dd-ff5192129be2/revoke-keys/0.log" Oct 30 22:13:20.410363 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:20.410335 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-db0fa3aa-5ktnz_137ef9a5-5d59-4c38-8890-fe71531e4d80/revoke-keys/0.log" Oct 30 22:13:20.592240 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:20.592207 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-jv74x_5230b38d-7e4a-4cb3-8071-6f3484fd4240/revoke-keys/0.log" Oct 30 22:13:21.884040 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:21.883998 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:22.026113 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:22.026079 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rxvkm"] Oct 30 22:13:22.029143 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:22.029111 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rxvkm"] Oct 30 22:13:22.168048 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:22.167965 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-fhf5g_fc623bd2-899d-4cfb-ac2b-4f6a8d0bd00d/manager/0.log" Oct 30 22:13:23.522159 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:23.522126 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f4fb010-f669-42d4-a654-91ca72cc9e82" path="/var/lib/kubelet/pods/0f4fb010-f669-42d4-a654-91ca72cc9e82/volumes" Oct 30 22:13:24.279399 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:24.279368 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5978df7f68-prrl7_56bc9daf-0942-46f1-aa51-49b8c868aa77/kube-auth-proxy/0.log" Oct 30 22:13:24.655093 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:24.655056 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-7wxsr_fda6dedb-779c-49d8-8fcf-ba31f73402ae/istio-proxy/0.log" Oct 30 22:13:24.837589 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:24.837555 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-8fqrl_57fe57d2-19a5-4ef7-b606-7753b0b7d902/istio-proxy/0.log" Oct 30 22:13:25.216656 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:25.216627 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-fm6jd_913cd5d1-0d49-41ba-921d-603a44b2126a/istio-proxy/0.log" Oct 30 22:13:25.392766 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:25.392718 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-hqmk4_c1e2cfd5-7f5a-4624-a753-30bc982288d4/istio-proxy/0.log" Oct 30 22:13:26.524625 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:26.524591 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-hm54n_006ecdc9-450d-4058-bd65-bebafca12ea0/payload-pre-processing/0.log" Oct 30 22:13:26.714258 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:26.714224 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-b87ts_06ff0bb8-b3e4-406a-a0b3-b20f27750e47/payload-processing/0.log" Oct 30 22:13:27.801860 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:27.801807 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-6vklf_7e0434b3-6ab0-42c5-9c5b-e4592daadca7/main/0.log" Oct 30 22:13:28.139822 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:28.139778 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bvwvv_b6307469-5f7d-41d9-9490-e8381ff9bbfb/main/0.log" Oct 30 22:13:28.506960 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:28.506860 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-tc92h_edecde9f-7b29-4563-bf29-bcb748233470/main/0.log" Oct 30 22:13:28.680543 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:28.680510 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-ttzg6_a8c67e3a-dafa-4ed4-b5e3-f1b341427687/main/0.log" Oct 30 22:13:30.276365 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:13:30.276325 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:13:37.734806 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:37.734744 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:37.844861 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:37.844811 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:39.026555 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:39.026516 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c8jdn"] Oct 30 22:13:39.029820 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:39.029775 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c8jdn"] Oct 30 22:13:39.522808 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:39.522775 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1124264-b8d0-4bde-b039-d3f9f68d5802" path="/var/lib/kubelet/pods/f1124264-b8d0-4bde-b039-d3f9f68d5802/volumes" Oct 30 22:13:59.024735 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:59.024696 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0fa3aa-5ktnz"] Oct 30 22:13:59.028943 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:59.028900 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0fa3aa-5ktnz"] Oct 30 22:13:59.434270 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:59.434235 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:59.522695 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:59.522666 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="137ef9a5-5d59-4c38-8890-fe71531e4d80" path="/var/lib/kubelet/pods/137ef9a5-5d59-4c38-8890-fe71531e4d80/volumes" Oct 30 22:13:59.583599 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:59.583571 5801 scope.go:122] "RemoveContainer" containerID="a7c6ea7e816d9706c59a1b480bc15eb0a161da5cddb8b43ba9a8f7ecc6e83b20" Oct 30 22:13:59.592139 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:13:59.592121 5801 scope.go:122] "RemoveContainer" containerID="c6cf0bc7b4c186a381613b4ccb76d3ab34a2a405ba26c0d3259eda341a06c519" Oct 30 22:14:00.277652 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:14:00.277612 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:14:08.499397 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:08.499269 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-b87ts_06ff0bb8-b3e4-406a-a0b3-b20f27750e47/payload-processing/0.log" Oct 30 22:14:08.672271 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:08.672241 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-b87ts_06ff0bb8-b3e4-406a-a0b3-b20f27750e47/payload-processing/0.log" Oct 30 22:14:09.610252 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:09.610207 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:09.714498 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:09.714456 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:30.277296 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:14:30.277254 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:14:35.934168 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:35.934115 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:36.034817 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:36.034789 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:58.489607 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:58.489573 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-b87ts_06ff0bb8-b3e4-406a-a0b3-b20f27750e47/payload-processing/0.log" Oct 30 22:14:58.679923 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:58.679887 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-b87ts_06ff0bb8-b3e4-406a-a0b3-b20f27750e47/payload-processing/0.log" Oct 30 22:14:59.004355 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:59.004318 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vrzqz/must-gather-m875m"] Oct 30 22:14:59.004786 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:59.004741 5801 cpu_manager.go:414] "containerMap: removing container" podUID="7894eb2b-44ba-4ae5-81dd-ff5192129be2" containerName="revoke-keys" Oct 30 22:14:59.004786 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:59.004776 5801 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7894eb2b-44ba-4ae5-81dd-ff5192129be2" containerName="revoke-keys" Oct 30 22:14:59.004950 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:59.004895 5801 memory_manager.go:365] "RemoveStaleState removing state" podUID="7894eb2b-44ba-4ae5-81dd-ff5192129be2" containerName="revoke-keys" Oct 30 22:14:59.008126 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:59.008095 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vrzqz/must-gather-m875m" Oct 30 22:14:59.008245 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:59.008094 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:59.013896 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:59.013876 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vrzqz\"/\"kube-root-ca.crt\"" Oct 30 22:14:59.014461 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:59.014445 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vrzqz\"/\"openshift-service-ca.crt\"" Oct 30 22:14:59.014569 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:59.014463 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vrzqz\"/\"default-dockercfg-p8sks\"" Oct 30 22:14:59.083284 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:59.083254 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7ea33ef8-8735-4360-89d6-73fb2a22047e-must-gather-output\") pod \"must-gather-m875m\" (UID: \"7ea33ef8-8735-4360-89d6-73fb2a22047e\") " pod="openshift-must-gather-vrzqz/must-gather-m875m" Oct 30 22:14:59.083485 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:59.083328 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncjjs\" (UniqueName: \"kubernetes.io/projected/7ea33ef8-8735-4360-89d6-73fb2a22047e-kube-api-access-ncjjs\") pod \"must-gather-m875m\" (UID: \"7ea33ef8-8735-4360-89d6-73fb2a22047e\") " pod="openshift-must-gather-vrzqz/must-gather-m875m" Oct 30 22:14:59.084222 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:59.084199 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vrzqz/must-gather-m875m"] Oct 30 22:14:59.121366 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:59.121343 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:59.184685 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:59.184660 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7ea33ef8-8735-4360-89d6-73fb2a22047e-must-gather-output\") pod \"must-gather-m875m\" (UID: \"7ea33ef8-8735-4360-89d6-73fb2a22047e\") " pod="openshift-must-gather-vrzqz/must-gather-m875m" Oct 30 22:14:59.184823 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:59.184715 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncjjs\" (UniqueName: \"kubernetes.io/projected/7ea33ef8-8735-4360-89d6-73fb2a22047e-kube-api-access-ncjjs\") pod \"must-gather-m875m\" (UID: \"7ea33ef8-8735-4360-89d6-73fb2a22047e\") " pod="openshift-must-gather-vrzqz/must-gather-m875m" Oct 30 22:14:59.185056 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:59.185038 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7ea33ef8-8735-4360-89d6-73fb2a22047e-must-gather-output\") pod \"must-gather-m875m\" (UID: \"7ea33ef8-8735-4360-89d6-73fb2a22047e\") " pod="openshift-must-gather-vrzqz/must-gather-m875m" Oct 30 22:14:59.197496 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:59.197470 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncjjs\" (UniqueName: \"kubernetes.io/projected/7ea33ef8-8735-4360-89d6-73fb2a22047e-kube-api-access-ncjjs\") pod \"must-gather-m875m\" (UID: \"7ea33ef8-8735-4360-89d6-73fb2a22047e\") " pod="openshift-must-gather-vrzqz/must-gather-m875m" Oct 30 22:14:59.322028 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:59.322011 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vrzqz/must-gather-m875m" Oct 30 22:14:59.496950 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:59.496902 5801 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:59.599223 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:59.599133 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vrzqz/must-gather-m875m"] Oct 30 22:14:59.601512 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:14:59.601488 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ea33ef8_8735_4360_89d6_73fb2a22047e.slice/crio-0c2e5da6b19fa5929c05743bb2413b53cdb9ef2bedd075e9ab3c1b46cbe68ee1 WatchSource:0}: Error finding container 0c2e5da6b19fa5929c05743bb2413b53cdb9ef2bedd075e9ab3c1b46cbe68ee1: Status 404 returned error can't find the container with id 0c2e5da6b19fa5929c05743bb2413b53cdb9ef2bedd075e9ab3c1b46cbe68ee1 Oct 30 22:14:59.603321 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:59.603304 5801 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:14:59.615247 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:59.615225 5801 scope.go:122] "RemoveContainer" containerID="eee5b63fc90eefb91cb9f0480fe34fc1ba5a5d639f8f8567e55dec6526c3e037" Oct 30 22:14:59.623598 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:59.623581 5801 scope.go:122] "RemoveContainer" containerID="fd7f277e4a0a587f788e966145e69f47f597723645468abbe5fe72acc3c0b5c7" Oct 30 22:14:59.631041 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:14:59.631022 5801 scope.go:122] "RemoveContainer" containerID="3e8ec8f72ce6f2f57bab4f563e4af478ccef48c55010a46e8a02e6a014831d20" Oct 30 22:15:00.163209 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:00.163175 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl"] Oct 30 22:15:00.166602 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:00.166578 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl" Oct 30 22:15:00.171442 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:00.171422 5801 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdvj7\"" Oct 30 22:15:00.172680 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:00.172657 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:15:00.172680 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:00.172660 5801 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:15:00.183169 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:00.183142 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl"] Oct 30 22:15:00.194175 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:00.194146 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ac8e94-32da-485e-a523-663ddf930061-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-v8cxl\" (UID: \"53ac8e94-32da-485e-a523-663ddf930061\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl" Oct 30 22:15:00.194282 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:00.194247 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc6cs\" (UniqueName: \"kubernetes.io/projected/53ac8e94-32da-485e-a523-663ddf930061-kube-api-access-lc6cs\") pod \"maas-api-key-cleanup-29889975-v8cxl\" (UID: \"53ac8e94-32da-485e-a523-663ddf930061\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl" Oct 30 22:15:00.281311 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:15:00.281278 5801 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-htw5p" containerName="extractor" Oct 30 22:15:00.294962 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:00.294936 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ac8e94-32da-485e-a523-663ddf930061-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-v8cxl\" (UID: \"53ac8e94-32da-485e-a523-663ddf930061\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl" Oct 30 22:15:00.295100 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:00.294979 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc6cs\" (UniqueName: \"kubernetes.io/projected/53ac8e94-32da-485e-a523-663ddf930061-kube-api-access-lc6cs\") pod \"maas-api-key-cleanup-29889975-v8cxl\" (UID: \"53ac8e94-32da-485e-a523-663ddf930061\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl" Oct 30 22:15:00.295699 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:00.295676 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ac8e94-32da-485e-a523-663ddf930061-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-v8cxl\" (UID: \"53ac8e94-32da-485e-a523-663ddf930061\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl" Oct 30 22:15:00.310589 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:00.310561 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc6cs\" (UniqueName: \"kubernetes.io/projected/53ac8e94-32da-485e-a523-663ddf930061-kube-api-access-lc6cs\") pod \"maas-api-key-cleanup-29889975-v8cxl\" (UID: \"53ac8e94-32da-485e-a523-663ddf930061\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl" Oct 30 22:15:00.461032 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:00.461004 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vrzqz/must-gather-m875m" event={"ID":"7ea33ef8-8735-4360-89d6-73fb2a22047e","Type":"ContainerStarted","Data":"0c2e5da6b19fa5929c05743bb2413b53cdb9ef2bedd075e9ab3c1b46cbe68ee1"} Oct 30 22:15:00.479898 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:00.479870 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl" Oct 30 22:15:00.778001 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:00.777977 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl"] Oct 30 22:15:00.780547 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:15:00.780511 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ac8e94_32da_485e_a523_663ddf930061.slice/crio-46b5ee7a35f622f1b7f03eb1ea3a2d6c8e788538c958d2f6f9017e0d9cddf2b2 WatchSource:0}: Error finding container 46b5ee7a35f622f1b7f03eb1ea3a2d6c8e788538c958d2f6f9017e0d9cddf2b2: Status 404 returned error can't find the container with id 46b5ee7a35f622f1b7f03eb1ea3a2d6c8e788538c958d2f6f9017e0d9cddf2b2 Oct 30 22:15:01.473530 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:01.473431 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl" event={"ID":"53ac8e94-32da-485e-a523-663ddf930061","Type":"ContainerStarted","Data":"e00d347c4785f616fa5bbe7690981624b30bfa94b2be256f03e50d5ccf4a5345"} Oct 30 22:15:01.473530 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:01.473492 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl" event={"ID":"53ac8e94-32da-485e-a523-663ddf930061","Type":"ContainerStarted","Data":"46b5ee7a35f622f1b7f03eb1ea3a2d6c8e788538c958d2f6f9017e0d9cddf2b2"} Oct 30 22:15:01.475861 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:01.475826 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vrzqz/must-gather-m875m" event={"ID":"7ea33ef8-8735-4360-89d6-73fb2a22047e","Type":"ContainerStarted","Data":"94378c463f3ee20edadb1400063229051af62e76ebcd64f30ae8e34f3fc94a36"} Oct 30 22:15:01.475861 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:01.475862 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vrzqz/must-gather-m875m" event={"ID":"7ea33ef8-8735-4360-89d6-73fb2a22047e","Type":"ContainerStarted","Data":"3b13c81bb2d7b783e0f9a86ac5d51074d55bcaef1f3681835068317d8c1b997a"} Oct 30 22:15:01.492742 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:01.492698 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl" podStartSLOduration=1.492687468 podStartE2EDuration="1.492687468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:01.490806647 +0000 UTC m=+2770.086623329" watchObservedRunningTime="2026-10-30 22:15:01.492687468 +0000 UTC m=+2770.088504127" Oct 30 22:15:01.513604 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:01.513264 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vrzqz/must-gather-m875m" podStartSLOduration=2.658713776 podStartE2EDuration="3.513248958s" totalImagesPullingTime="854.535182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:01.511277273 +0000 UTC m=+2770.107093934" watchObservedRunningTime="2026-10-30 22:15:01.513248958 +0000 UTC m=+2770.109065619" Oct 30 22:15:02.481629 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:02.481596 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-c2vrg_183318eb-952e-4e93-b716-6ab50b9c58dd/global-pull-secret-syncer/0.log" Oct 30 22:15:02.484323 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:02.484297 5801 generic.go:356] "Generic (PLEG): container finished" podID="53ac8e94-32da-485e-a523-663ddf930061" containerID="e00d347c4785f616fa5bbe7690981624b30bfa94b2be256f03e50d5ccf4a5345" exitCode=60 Oct 30 22:15:02.484476 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:02.484381 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl" event={"ID":"53ac8e94-32da-485e-a523-663ddf930061","Type":"ContainerDied","Data":"e00d347c4785f616fa5bbe7690981624b30bfa94b2be256f03e50d5ccf4a5345"} Oct 30 22:15:02.484614 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:02.484599 5801 scope.go:122] "RemoveContainer" containerID="e00d347c4785f616fa5bbe7690981624b30bfa94b2be256f03e50d5ccf4a5345" Oct 30 22:15:02.809023 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:02.808991 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7d6nt_950f0225-e1c9-4c9c-8fc6-25dba5db7120/konnectivity-agent/0.log" Oct 30 22:15:02.977078 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:02.977046 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal_3b404058373133d538844156a64ac398/haproxy/0.log" Oct 30 22:15:03.498882 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:03.497477 5801 generic.go:356] "Generic (PLEG): container finished" podID="53ac8e94-32da-485e-a523-663ddf930061" containerID="6b5318ba6a034e0e8aa06a16fca46fbd1731f8c4dfd8b1eb83d3413898cd5786" exitCode=60 Oct 30 22:15:03.498882 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:03.497544 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl" event={"ID":"53ac8e94-32da-485e-a523-663ddf930061","Type":"ContainerDied","Data":"6b5318ba6a034e0e8aa06a16fca46fbd1731f8c4dfd8b1eb83d3413898cd5786"} Oct 30 22:15:03.498882 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:03.497601 5801 scope.go:122] "RemoveContainer" containerID="e00d347c4785f616fa5bbe7690981624b30bfa94b2be256f03e50d5ccf4a5345" Oct 30 22:15:03.498882 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:03.498079 5801 scope.go:122] "RemoveContainer" containerID="6b5318ba6a034e0e8aa06a16fca46fbd1731f8c4dfd8b1eb83d3413898cd5786" Oct 30 22:15:03.498882 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:15:03.498398 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-v8cxl_odh-ai-gateway-infra(53ac8e94-32da-485e-a523-663ddf930061)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl" podUID="53ac8e94-32da-485e-a523-663ddf930061" Oct 30 22:15:04.533198 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:04.533170 5801 scope.go:122] "RemoveContainer" containerID="6b5318ba6a034e0e8aa06a16fca46fbd1731f8c4dfd8b1eb83d3413898cd5786" Oct 30 22:15:04.534399 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:15:04.534367 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-v8cxl_odh-ai-gateway-infra(53ac8e94-32da-485e-a523-663ddf930061)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl" podUID="53ac8e94-32da-485e-a523-663ddf930061" Oct 30 22:15:09.568484 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:09.568433 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-fhf5g_fc623bd2-899d-4cfb-ac2b-4f6a8d0bd00d/manager/0.log" Oct 30 22:15:11.706768 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:11.706720 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h847d_21593b07-35ab-409e-98ef-74167147e27c/node-exporter/0.log" Oct 30 22:15:11.735295 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:11.735259 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h847d_21593b07-35ab-409e-98ef-74167147e27c/kube-rbac-proxy/0.log" Oct 30 22:15:11.756880 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:11.755742 5801 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6"] Oct 30 22:15:11.777962 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:11.777930 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6"] Oct 30 22:15:11.778294 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:11.778243 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6" Oct 30 22:15:11.787551 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:11.787529 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h847d_21593b07-35ab-409e-98ef-74167147e27c/init-textfile/0.log" Oct 30 22:15:11.826532 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:11.826479 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3ab766d-bcda-4a26-89e8-1dc9ad805fe0-sys\") pod \"perf-node-gather-daemonset-m8jl6\" (UID: \"a3ab766d-bcda-4a26-89e8-1dc9ad805fe0\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6" Oct 30 22:15:11.826723 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:11.826542 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmw9h\" (UniqueName: \"kubernetes.io/projected/a3ab766d-bcda-4a26-89e8-1dc9ad805fe0-kube-api-access-rmw9h\") pod \"perf-node-gather-daemonset-m8jl6\" (UID: \"a3ab766d-bcda-4a26-89e8-1dc9ad805fe0\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6" Oct 30 22:15:11.826723 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:11.826661 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a3ab766d-bcda-4a26-89e8-1dc9ad805fe0-lib-modules\") pod \"perf-node-gather-daemonset-m8jl6\" (UID: \"a3ab766d-bcda-4a26-89e8-1dc9ad805fe0\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6" Oct 30 22:15:11.826723 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:11.826697 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a3ab766d-bcda-4a26-89e8-1dc9ad805fe0-podres\") pod \"perf-node-gather-daemonset-m8jl6\" (UID: \"a3ab766d-bcda-4a26-89e8-1dc9ad805fe0\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6" Oct 30 22:15:11.826897 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:11.826871 5801 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a3ab766d-bcda-4a26-89e8-1dc9ad805fe0-proc\") pod \"perf-node-gather-daemonset-m8jl6\" (UID: \"a3ab766d-bcda-4a26-89e8-1dc9ad805fe0\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6" Oct 30 22:15:11.918319 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:11.918281 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a39962c3-e95b-4c79-bba1-f0c536c3575f/prometheus/0.log" Oct 30 22:15:11.928413 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:11.928385 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a3ab766d-bcda-4a26-89e8-1dc9ad805fe0-lib-modules\") pod \"perf-node-gather-daemonset-m8jl6\" (UID: \"a3ab766d-bcda-4a26-89e8-1dc9ad805fe0\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6" Oct 30 22:15:11.928556 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:11.928416 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a3ab766d-bcda-4a26-89e8-1dc9ad805fe0-podres\") pod \"perf-node-gather-daemonset-m8jl6\" (UID: \"a3ab766d-bcda-4a26-89e8-1dc9ad805fe0\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6" Oct 30 22:15:11.928556 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:11.928474 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a3ab766d-bcda-4a26-89e8-1dc9ad805fe0-proc\") pod \"perf-node-gather-daemonset-m8jl6\" (UID: \"a3ab766d-bcda-4a26-89e8-1dc9ad805fe0\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6" Oct 30 22:15:11.928556 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:11.928524 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3ab766d-bcda-4a26-89e8-1dc9ad805fe0-sys\") pod \"perf-node-gather-daemonset-m8jl6\" (UID: \"a3ab766d-bcda-4a26-89e8-1dc9ad805fe0\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6" Oct 30 22:15:11.928556 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:11.928527 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a3ab766d-bcda-4a26-89e8-1dc9ad805fe0-lib-modules\") pod \"perf-node-gather-daemonset-m8jl6\" (UID: \"a3ab766d-bcda-4a26-89e8-1dc9ad805fe0\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6" Oct 30 22:15:11.928747 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:11.928554 5801 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmw9h\" (UniqueName: \"kubernetes.io/projected/a3ab766d-bcda-4a26-89e8-1dc9ad805fe0-kube-api-access-rmw9h\") pod \"perf-node-gather-daemonset-m8jl6\" (UID: \"a3ab766d-bcda-4a26-89e8-1dc9ad805fe0\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6" Oct 30 22:15:11.928747 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:11.928596 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a3ab766d-bcda-4a26-89e8-1dc9ad805fe0-podres\") pod \"perf-node-gather-daemonset-m8jl6\" (UID: \"a3ab766d-bcda-4a26-89e8-1dc9ad805fe0\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6" Oct 30 22:15:11.928747 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:11.928607 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a3ab766d-bcda-4a26-89e8-1dc9ad805fe0-proc\") pod \"perf-node-gather-daemonset-m8jl6\" (UID: \"a3ab766d-bcda-4a26-89e8-1dc9ad805fe0\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6" Oct 30 22:15:11.928747 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:11.928637 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3ab766d-bcda-4a26-89e8-1dc9ad805fe0-sys\") pod \"perf-node-gather-daemonset-m8jl6\" (UID: \"a3ab766d-bcda-4a26-89e8-1dc9ad805fe0\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6" Oct 30 22:15:11.940944 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:11.940918 5801 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmw9h\" (UniqueName: \"kubernetes.io/projected/a3ab766d-bcda-4a26-89e8-1dc9ad805fe0-kube-api-access-rmw9h\") pod \"perf-node-gather-daemonset-m8jl6\" (UID: \"a3ab766d-bcda-4a26-89e8-1dc9ad805fe0\") " pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6" Oct 30 22:15:11.943085 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:11.943062 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a39962c3-e95b-4c79-bba1-f0c536c3575f/config-reloader/0.log" Oct 30 22:15:11.986197 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:11.986114 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a39962c3-e95b-4c79-bba1-f0c536c3575f/thanos-sidecar/0.log" Oct 30 22:15:12.012652 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:12.012617 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a39962c3-e95b-4c79-bba1-f0c536c3575f/kube-rbac-proxy-web/0.log" Oct 30 22:15:12.039169 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:12.039137 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a39962c3-e95b-4c79-bba1-f0c536c3575f/kube-rbac-proxy/0.log" Oct 30 22:15:12.066065 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:12.066038 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a39962c3-e95b-4c79-bba1-f0c536c3575f/kube-rbac-proxy-thanos/0.log" Oct 30 22:15:12.092893 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:12.092864 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a39962c3-e95b-4c79-bba1-f0c536c3575f/init-config-reloader/0.log" Oct 30 22:15:12.100615 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:12.100591 5801 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6" Oct 30 22:15:12.135574 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:12.135541 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-xh9wc_7f1829b1-8087-46b1-b64d-e3f15811caf6/prometheus-operator/0.log" Oct 30 22:15:12.164683 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:12.164651 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-xh9wc_7f1829b1-8087-46b1-b64d-e3f15811caf6/kube-rbac-proxy/0.log" Oct 30 22:15:12.230696 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:12.230671 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d84c6fb95-spv7p_da5c17e9-1229-4d33-ba12-3eab4727e64d/telemeter-client/0.log" Oct 30 22:15:12.256564 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:12.256481 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d84c6fb95-spv7p_da5c17e9-1229-4d33-ba12-3eab4727e64d/reload/0.log" Oct 30 22:15:12.284138 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:12.284107 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d84c6fb95-spv7p_da5c17e9-1229-4d33-ba12-3eab4727e64d/kube-rbac-proxy/0.log" Oct 30 22:15:12.409271 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:12.409235 5801 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6"] Oct 30 22:15:12.414267 ip-10-0-140-163 kubenswrapper[5801]: W1030 22:15:12.414240 5801 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda3ab766d_bcda_4a26_89e8_1dc9ad805fe0.slice/crio-e5e50ce5dbff49fef01fb5e3c9c361bb1b9d2113487891f368aee923482741e7 WatchSource:0}: Error finding container e5e50ce5dbff49fef01fb5e3c9c361bb1b9d2113487891f368aee923482741e7: Status 404 returned error can't find the container with id e5e50ce5dbff49fef01fb5e3c9c361bb1b9d2113487891f368aee923482741e7 Oct 30 22:15:12.637911 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:12.637866 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6" event={"ID":"a3ab766d-bcda-4a26-89e8-1dc9ad805fe0","Type":"ContainerStarted","Data":"fd426a203ac203ba18707bbf8143e45ca7ad53e7e9bbcdf3b4ea5191897bf409"} Oct 30 22:15:12.638121 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:12.637978 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6" event={"ID":"a3ab766d-bcda-4a26-89e8-1dc9ad805fe0","Type":"ContainerStarted","Data":"e5e50ce5dbff49fef01fb5e3c9c361bb1b9d2113487891f368aee923482741e7"} Oct 30 22:15:12.638238 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:12.638214 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6" Oct 30 22:15:12.658457 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:12.658410 5801 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6" podStartSLOduration=1.658384493 podStartE2EDuration="1.658384493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:12.656177062 +0000 UTC m=+2781.251993723" watchObservedRunningTime="2026-10-30 22:15:12.658384493 +0000 UTC m=+2781.254201153" Oct 30 22:15:16.516850 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:16.516816 5801 scope.go:122] "RemoveContainer" containerID="6b5318ba6a034e0e8aa06a16fca46fbd1731f8c4dfd8b1eb83d3413898cd5786" Oct 30 22:15:16.635418 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:16.635381 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-99chm_5c702aef-eccc-4da1-8040-c860c607a466/dns/0.log" Oct 30 22:15:16.660867 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:16.660840 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-99chm_5c702aef-eccc-4da1-8040-c860c607a466/kube-rbac-proxy/0.log" Oct 30 22:15:16.840539 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:16.840514 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j2j9f_c3af705e-b3ba-4379-8214-07ba1b431824/dns-node-resolver/0.log" Oct 30 22:15:17.629276 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:17.629188 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-db7tz_5934c36a-2955-4f0c-a8e8-615f2fa07454/node-ca/0.log" Oct 30 22:15:17.676793 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:17.673626 5801 generic.go:356] "Generic (PLEG): container finished" podID="53ac8e94-32da-485e-a523-663ddf930061" containerID="c5b93e27692c3fc1237b5336cfe5e65070cc35b6a989d244c3a8434bd121aa76" exitCode=60 Oct 30 22:15:17.676793 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:17.673853 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl" event={"ID":"53ac8e94-32da-485e-a523-663ddf930061","Type":"ContainerDied","Data":"c5b93e27692c3fc1237b5336cfe5e65070cc35b6a989d244c3a8434bd121aa76"} Oct 30 22:15:17.676793 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:17.673918 5801 scope.go:122] "RemoveContainer" containerID="6b5318ba6a034e0e8aa06a16fca46fbd1731f8c4dfd8b1eb83d3413898cd5786" Oct 30 22:15:17.676793 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:17.674382 5801 scope.go:122] "RemoveContainer" containerID="c5b93e27692c3fc1237b5336cfe5e65070cc35b6a989d244c3a8434bd121aa76" Oct 30 22:15:17.676793 ip-10-0-140-163 kubenswrapper[5801]: E1030 22:15:17.674685 5801 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-v8cxl_odh-ai-gateway-infra(53ac8e94-32da-485e-a523-663ddf930061)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl" podUID="53ac8e94-32da-485e-a523-663ddf930061" Oct 30 22:15:18.660842 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:18.660817 5801 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vrzqz/perf-node-gather-daemonset-m8jl6" Oct 30 22:15:18.704739 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:18.704698 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl"] Oct 30 22:15:18.886698 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:18.886673 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl" Oct 30 22:15:19.034427 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:19.033645 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53ac8e94-32da-485e-a523-663ddf930061-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ac8e94-32da-485e-a523-663ddf930061-maas-api-service-ca\") pod \"53ac8e94-32da-485e-a523-663ddf930061\" (UID: \"53ac8e94-32da-485e-a523-663ddf930061\") " Oct 30 22:15:19.034427 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:19.033702 5801 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53ac8e94-32da-485e-a523-663ddf930061-kube-api-access-lc6cs\" (UniqueName: \"kubernetes.io/projected/53ac8e94-32da-485e-a523-663ddf930061-kube-api-access-lc6cs\") pod \"53ac8e94-32da-485e-a523-663ddf930061\" (UID: \"53ac8e94-32da-485e-a523-663ddf930061\") " Oct 30 22:15:19.034427 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:19.034117 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ac8e94-32da-485e-a523-663ddf930061-maas-api-service-ca" pod "53ac8e94-32da-485e-a523-663ddf930061" (UID: "53ac8e94-32da-485e-a523-663ddf930061"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:19.034728 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:19.034571 5801 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ac8e94-32da-485e-a523-663ddf930061-maas-api-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:19.036766 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:19.036728 5801 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ac8e94-32da-485e-a523-663ddf930061-kube-api-access-lc6cs" pod "53ac8e94-32da-485e-a523-663ddf930061" (UID: "53ac8e94-32da-485e-a523-663ddf930061"). InnerVolumeSpecName "kube-api-access-lc6cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:19.135920 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:19.135887 5801 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc6cs\" (UniqueName: \"kubernetes.io/projected/53ac8e94-32da-485e-a523-663ddf930061-kube-api-access-lc6cs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:19.691815 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:19.691774 5801 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl" event={"ID":"53ac8e94-32da-485e-a523-663ddf930061","Type":"ContainerDied","Data":"46b5ee7a35f622f1b7f03eb1ea3a2d6c8e788538c958d2f6f9017e0d9cddf2b2"} Oct 30 22:15:19.692383 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:19.691838 5801 scope.go:122] "RemoveContainer" containerID="c5b93e27692c3fc1237b5336cfe5e65070cc35b6a989d244c3a8434bd121aa76" Oct 30 22:15:19.692383 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:19.691790 5801 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl" Oct 30 22:15:19.713473 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:19.713444 5801 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl"] Oct 30 22:15:19.717199 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:19.717172 5801 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-v8cxl"] Oct 30 22:15:20.470792 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:20.470138 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5978df7f68-prrl7_56bc9daf-0942-46f1-aa51-49b8c868aa77/kube-auth-proxy/0.log" Oct 30 22:15:20.547779 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:20.547733 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-7wxsr_fda6dedb-779c-49d8-8fcf-ba31f73402ae/istio-proxy/0.log" Oct 30 22:15:20.578829 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:20.578798 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-8fqrl_57fe57d2-19a5-4ef7-b606-7753b0b7d902/istio-proxy/0.log" Oct 30 22:15:20.648454 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:20.648416 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-fm6jd_913cd5d1-0d49-41ba-921d-603a44b2126a/istio-proxy/0.log" Oct 30 22:15:20.684502 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:20.684470 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-hqmk4_c1e2cfd5-7f5a-4624-a753-30bc982288d4/istio-proxy/0.log" Oct 30 22:15:20.914440 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:20.914409 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-hm54n_006ecdc9-450d-4058-bd65-bebafca12ea0/payload-pre-processing/0.log" Oct 30 22:15:20.951775 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:20.951149 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-b87ts_06ff0bb8-b3e4-406a-a0b3-b20f27750e47/payload-processing/0.log" Oct 30 22:15:21.526640 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:21.526607 5801 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53ac8e94-32da-485e-a523-663ddf930061" path="/var/lib/kubelet/pods/53ac8e94-32da-485e-a523-663ddf930061/volumes" Oct 30 22:15:23.039218 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:23.039192 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wdr8m_808521e6-a0b3-4a24-b0b0-5a3cabffe0f5/serve-healthcheck-canary/0.log" Oct 30 22:15:23.370819 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:23.370792 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5978df7f68-prrl7_56bc9daf-0942-46f1-aa51-49b8c868aa77/kube-auth-proxy/0.log" Oct 30 22:15:23.418065 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:23.418016 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-7wxsr_fda6dedb-779c-49d8-8fcf-ba31f73402ae/istio-proxy/0.log" Oct 30 22:15:23.450712 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:23.450682 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-8fqrl_57fe57d2-19a5-4ef7-b606-7753b0b7d902/istio-proxy/0.log" Oct 30 22:15:23.495744 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:23.495714 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-fm6jd_913cd5d1-0d49-41ba-921d-603a44b2126a/istio-proxy/0.log" Oct 30 22:15:23.524116 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:23.524085 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-hqmk4_c1e2cfd5-7f5a-4624-a753-30bc982288d4/istio-proxy/0.log" Oct 30 22:15:23.663456 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:23.663383 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-hm54n_006ecdc9-450d-4058-bd65-bebafca12ea0/payload-pre-processing/0.log" Oct 30 22:15:23.681575 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:23.681546 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-b87ts_06ff0bb8-b3e4-406a-a0b3-b20f27750e47/payload-processing/0.log" Oct 30 22:15:23.769804 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:23.769708 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-htw5p_a31aa40d-e199-4095-b9b9-f2c7e31822e2/kube-rbac-proxy/0.log" Oct 30 22:15:23.797041 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:23.797016 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-htw5p_a31aa40d-e199-4095-b9b9-f2c7e31822e2/exporter/0.log" Oct 30 22:15:23.827852 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:23.827824 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-htw5p_a31aa40d-e199-4095-b9b9-f2c7e31822e2/extractor/0.log" Oct 30 22:15:26.539617 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:26.539573 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6bbc596d7f-nqfhb_5b8871bf-6ac4-4d11-a064-f0a762af556d/manager/0.log" Oct 30 22:15:26.560341 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:26.560315 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6bbc596d7f-nqfhb_5b8871bf-6ac4-4d11-a064-f0a762af556d/copy-manifests/0.log" Oct 30 22:15:27.160209 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:27.160175 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5978df7f68-prrl7_56bc9daf-0942-46f1-aa51-49b8c868aa77/kube-auth-proxy/0.log" Oct 30 22:15:27.207079 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:27.207048 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-7wxsr_fda6dedb-779c-49d8-8fcf-ba31f73402ae/istio-proxy/0.log" Oct 30 22:15:27.225672 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:27.225642 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-8fqrl_57fe57d2-19a5-4ef7-b606-7753b0b7d902/istio-proxy/0.log" Oct 30 22:15:27.266717 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:27.266687 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-fm6jd_913cd5d1-0d49-41ba-921d-603a44b2126a/istio-proxy/0.log" Oct 30 22:15:27.286148 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:27.286102 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-hqmk4_c1e2cfd5-7f5a-4624-a753-30bc982288d4/istio-proxy/0.log" Oct 30 22:15:27.406605 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:27.406574 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-hm54n_006ecdc9-450d-4058-bd65-bebafca12ea0/payload-pre-processing/0.log" Oct 30 22:15:27.425260 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:27.425184 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-b87ts_06ff0bb8-b3e4-406a-a0b3-b20f27750e47/payload-processing/0.log" Oct 30 22:15:27.747162 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:27.747080 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-5c988cdb58-dr7v5_b6ba8f2e-a8c3-4cdc-8cb4-9716a4c6a388/manager/0.log" Oct 30 22:15:27.778797 ip-10-0-140-163 kubenswrapper[5801]: I1030 22:15:27.778730 5801 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-mwmbn_296ecffb-f5b2-4152-8b81-d005552fe028/jobset-operator/0.log"